Using trepan With A Multithreaded Application

[jterk]

I'm trying to use trepan to debug a multithreaded web application remotely. I've managed to get remotely connected by doing this on the server:

from threading import Thread
from trepan import debugger
from trepan.api import debug
from trepan.interfaces import server

def _trepan_run():
    connection_opts = {
        'IO': 'TCP',
        'PORT': 1027
    }

    interface = server.ServerInterface(connection_opts=connection_opts)

    debug_opts = {
        'interface': interface
    }

    debug(dbg_opts=debug_opts)

    # An alternative attempt to allow the client to attach
    #
    # dbg = debugger.Debugger(debug_opts)
    # dbg.core.add_ignore(_trepan_run)
    # dbg.core.start()
    #
    # while True:
    #     dbg.core.processor.process_commands()

t = Thread(target=_trepan_run)
t.daemon = True
t.start()

I'm able to connect using python /usr/local/lib/python2.7/site-packages/trepan/client.py -H HOSTNAME and I seem to be able to set breakpoints (I get a Breakpoint 1 set at line X of file FILE message), but execution never stops on the breakpoints I set.

Can you provide any guidance on whether my use-case is possible with trepan and, if so, what I need to do to make it work?

[rocky]

Here's what's going on. You are starting the debugger in a thread, so I believe it it is only able to trace events in that thread and threads spawned by thread. To see this, when you are stopped inside the debugger run info threads to see the threads available and issue the command set trace on and you'll probably only see trace events for that thread you are stopped in. (I should fix the trace display to show the thread number when there's more than one)

A possible workaround is to enter the debugger from a signal handler as described in https://github.com/rocky/python2-trepan/wiki/Getting-into-the-Debugger#set-up-an-exception-handler-to-enter-the-debugger-on-a-signal

Here is some code that merges that with your code:

from threading import Thread
from trepan.api import debug
from trepan.interfaces import server
import time

import signal
def signal_handler(num, f):
    connection_opts = {
        'IO': 'TCP',
        'PORT': 1027
    }

    interface = server.ServerInterface(connection_opts=connection_opts)

    debug_opts = {
        'interface': interface
    }

    debug(dbg_opts=debug_opts)
    return
signal.signal(signal.SIGUSR1, signal_handler)

def my_thread():
    for i in range(1000):
        z = i % 2
        time.sleep(z)

import os
print("PID %d" % os.getpid())
t = Thread(target=my_thread)
t.daemon = True
t.start()
for i in range(1000):
    q = i % 5
    time.sleep(1)

[rocky]

I should also say that if you find something that works for you here, please at it to the wiki for this project.

[jterk]

I'll re-open if I make any headway.

[jterk]

I was able to get this to work - the core issue is that trepan only configures tracer to register a trace function on the current thread (it doesn't supply the 'include-threads' argument to tracer).

I was able to get around this without modifying trepan by adding the following (as early on in the program as possible, since there's no mechanism for tracing an already extant thread):

from trepan import debugger
from trepan.api import debug
from trepan.interfaces import server
from trepan.lib import default

default.START_OPTS['include_threads'] = True

connection_opts = {
    'IO': 'TCP',
    'PORT': 1027
}

interface = server.ServerInterface(connection_opts=connection_opts)

debug_opts = {
    'interface': interface,
}

dbg = debugger.Debugger(debug_opts)
dbg.core.start()

It's worth noting that this necessarily suspends execution until one connects the debug client and issues a command to resume execution.

[rocky]

Thanks for investigating, reporting, and offering a workaround.

We can change trepan2 though so you don't have to work around. One proposal is to add a key, like tracer_start_opts to the opts parameter of core.start which will get merge in with default options.

Would that work? Or do you have another proposal?

Also, we should document how to do this so others can more easily follow in your footsteps. Would you care to offer up documentation changes or suggestions of where we should describe the situation and the consequences of this approach? One suggestion is the doc on Entering the debugger

Thanks in advance for the help.