[ Previous | Next | Table of Contents | Index | Library Home | Legal | Search ]

General Programming Concepts: Writing and Debugging Programs


Developing Multi-Threaded Program Debuggers

The pthread debug library (libpthdebug.a) provides a set of functions which will allow debugger developers to provide debug capabilities for applications using the pthread library.

This library is used to debug both 32-bit and 64-bit pthreaded applications. This library is used to debug targeted debug processes only, it can also be used introspectively (for example: linked to an application that uses pthreads) to examine pthread information of its own application. See Developing Multi-Threaded Program which examines and modifies pthread library objects. This library can be used by a multi-threaded debugger to debug a multi-threaded application. Multi-threaded debuggers are supported via libpthreads.a. This library is thread safe. The pthread debug library contains a 32-bit shared object and a 64-bit shared object.

Debuggers using the ptrace facility must link to the 32-bit version of the library, since the ptrace facility is not suppoted in 64-bit mode. Debuggers using the /proc facility can link to either the 32-bit version or the 64-bit version of this library.

The pthread debug library provides debuggers access to pthread library information. This includes information on pthreads, pthread attributes, mutexes, mutex attributes, condition variables, condition variable attributes, read/write locks, read/write lock attributes, and information about the state of the pthread library. This library also provides help with controlling the execution of pthreads.

Note: All data (addresses, registers) returned by this library will be in 64-bit format both for 64-bit and 32-bit application. It is the debuggers responsibility to convert these values into 32-bit format for 32-bit applications. When debugging a 32-bit application the top half of addresses and registers will be ignored.

Note: The pthread debug library does not report mutexes, mutexattrs, conds, condattrs, rwlocks, rwlockattrs that have the pshared value of PTHREAD_PROCESS_SHARED.

Initialization

The debugger must initialize a pthread debug library session for each debug process. This cannot be done until the pthread library has been initialized in the debug process. The pthdb_session_pthreaded() function has been provided to tell the debugger when the pthread library has been initialized in the debug process. Each time, the pthdb_session_pthreaded() function is called it checks to see if the pthread library has been initialized. If initialized, it returns PTHDB_SUCCESS. Otherwise it returns PTHDB_NOT_PTHREADED. In both cases, it returns a function name which can be used to set a breakpoint for immediate notification that the pthread library has been initialized. Therefore, the pthdb_session_pthreaded() function provides two methods to determine when the pthread library has been initialized:

Once the debug process is pthreaded, the debugger must call the pthdb_session_init() function, to initialize a session for the debug process. The pthread debug library supports one session for a single debug process. The debugger must assign a unique user identifier and pass it to pthdb_session_init() which in turn will assign a unique session identifier which must be passed as the first parameter to all other pthread debug library functions, except pthdb_session_pthreaded(), in return. Whenever the pthread debug library invokes a Multi-Threaded Call Back Functions, it will pass the unique debugger assigned user identifier back to the debugger. The pthdb_session_init() function checks the list of call back functions provided by the debugger, and initializes the session's data structures. Also, this function sets the session flags, see the pthdb_session_setflags function.

Call Back Functions

The pthread debug library uses call back functions to obtain addresses, to obtain data, to write data, to give storage management to the debugger, and to aid in debugging the pthread debug library. See Multi-Threaded Call Back Functions for more information.

Update Function

Each time the debugger is stopped, after the session has been initialized, it is necessary to call the pthdb_session_update() function. This function sets or reset the lists of pthreads, pthread attributes, mutexes, mutex attributes, condition variables, condition variable attributes, read/write locks, read/write lock attributes, pthread specific keys and active keys. It uses call back functions to manage memory for the lists.

Hold and Unhold Functions

Debuggers need to support hold and unhold of threads for two reasons:

The pthdb_pthread_hold() function sets the hold state of a pthread to hold.

The pthdb_pthread_unhold() function sets the hold state of a pthread to unhold.

Note: The pthdb_pthread_hold() and pthdb_pthread_unhold() functions must always be used whether a pthread has a kernel thread or not.

The pthdb_pthread_holdstate() function returns the hold state of the pthread.

The pthdb_session_committed() function reports the function name of the function that is called after all of the hold and unhold changes are committed. A break point can be placed at this function to notify the debugger when the hold and unhold changes have been committed.

The pthdb_session_stop_tid() function informs the pthread debug library, which informs the pthread library the tid of the thread that stopped the debugger.

The pthdb_session_commit_tid() function returns the list of kernel threads, one kernel thread at a time, that must be continued to commit the hold and unhold changes. This function must be called repeatedly, until PTHDB_INVALID_TID is reported. If the list of kernel threads is empty, it is not necessary to continue any threads for the commit operation.

The debugger can determine when all of the hold and unhold changes have been committed in two ways:

In order to hold or unhold pthreads it is necessary to follow the following procedure, before continuing a group of pthreads or single stepping a single pthread:

  1. Use the pthdb_pthread_hold() and pthdb_pthread_unhold() functions to set up which pthreads will be held and which will be unheld.
  2. Set-up the method that will determine when all of the hold and unhold changes have been committed.
  3. Use the pthdb_session_commit_tid() function to determine the list of tids that must be continued to commit the hold and unhold changes.
  4. Continue the tids in the previous step and the thread which stopped the debugger.

The pthdb_session_continue_tid() function allows the debugger to obtain the list of kernel threads that must be continued before it proceeds with single stepping a single pthread or continuing a group of pthreads. This function must be called repeatedly, until PTHDB_INVALID_TID is reported. If the list of kernel threads is not empty, the debugger will need to continue these kernel threads along with the others it is explicitly interested in. The debugger is responsible for parking the stop thread and continuing the stop thread. The stop thread, is the thread that caused the debugger to be entered.

Context Functions

The pthdb_pthread_context() function is used to get the context information and the pthdb_pthread_setcontext() function is used to set the context. The pthdb_pthread_context() function obtains the context information of a pthread from either the kernel or the pthread data structure in the debug process's address space. If the pthread is not associated with a kernel thread, then the context information saved by pthread library is obtained. If a pthread is associated with a kernel thread, the information is obtained from the debugger using call backs, it is the debuggers responsibility to determine if the kernel thread is in kernel mode or user mode and provide the correct information for that mode.

When a pthread with kernel thread is in kernel mode code it is impossible to get the full user mode context because the kernel does not save it off in one place. The getthrds() function can be used to get part of this information. It always saves the user mode stack and the debugger can discover this by checking thrdsinfo64.ti_scount. If this is non-zero the user mode stack is available in thrdsinfo64.ti_ustk. From user mode stack it is possible to determine the iar and the call back frames but not the other register values. The thrdsinfo64 structure is defined in procinfo.h file.

List Functions

The pthread debug library maintains lists for pthreads, pthread attributes, mutexes, mutex attributes, condition variables, condition variables attributes, read/write locks, read/write lock attributes, pthread specific keys and active keys, each represented by a type specific handle. The pthdb_<object>() The pthdb_<object>() functions return the next handle in the appropriate list, where <object> is one of the following: pthread, attr, mutex, mutexattr, cond, condattr, rwlock, rwlockattr or key. If the list is empty or the end of the list is reached, PTHDB_INVALID_object is reported, where object is one of the following: PTHREAD, ATTR, MUTEX, MUTEXATTR, COND, CONDATTR, RWLOCK, RWLOCKATTR or KEY.

Field Functions

Detailed information about an object can be obtained by using the appropriate object member function, pthdb_object_field(), where object is one of the following: pthread, attr, mutex, mutexattr, cond, condattr, rwlock, rwlockattr or key and where field is the name of a field of the detailed information for the object.

Customizing the Session

The pthdb_session_setflags() function allows the debugger to change the flags which customize the session. These flags are used to control the number of registers that are read or wrote during context operations, and to control the printing of debug information.

The pthdb_session_flags() function gets the current flags for the session.

Session Termination

At the end of the debug session, the session data structures need to be deallocated and the session data needs to be deleted. This is accomplished by calling the pthdb_session_destroy() function, which uses a call back functions to deallocate the memory. All of the memory allocated by the pthdb_session_init(), and pthdb_session_update() functions will be deallocated.

Example

Pseudo-code showing how the debugger should make use of the hold/unhold code:

/* includes */
 
#include <sys/pthdebug.h>
 
main()
{
    tid_t stop_tid; /* thread which stopped the process */
    pthdb_user_t user = <unique debugger value>;
    pthdb_session_t session; /* <unique library value> */
    pthdb_callbacks_t callbacks = <callback functions>;
    char *pthreaded_symbol=NULL;
    char *committed_symbol;
    int pthreaded = 0;
    int pthdb_init = 0;
    char *committed_symbol;
 
    /* fork/exec or attach to debuggee */
 
    /* debuggee uses ptrace()/ptracex() with PT_TRACE_ME */
 
    while (/* waiting on an event */) 
    {
      /* debugger waits on debuggee */
 
      if (pthreaded_symbol==NULL) {
        rc = pthdb_session_pthreaded(user, &callbacks, pthreaded_symbol);
        if (rc == PTHDB_NOT_PTHREADED)
        {
            /* set breakpoint at pthreaded_symbol */
        }
        else
          pthreaded=1;	
      }
      if (pthreaded == 1 && pthdb_init == 0) {
          rc = pthdb_session_init(user, &session, PEM_32BIT, flags, &callbacks);
          if (rc)
              /* handle error and exit */
          pthdb_init=1;
      }
  
      rc = pthdb_session_update(session)
      if ( rc != PTHDB_SUCCESS)
	/* handle error and exit */
 
      while (/* accepting debugger commands */)
      {
          switch (/* debugger command */)
          {
              ...
              case DB_HOLD:
                  /* regardless of pthread with or without kernel thread */
                  rc = pthdb_pthread_hold(session, pthread);
                  if (rc)
                      /* handle error and exit */
              case DB_UNHOLD:
                  /* regardless of pthread with or without kernel thread */
                  rc = pthdb_pthread_unhold(session, pthread);
                  if (rc)
                      /* handle error and exit */
              case DB_CONTINUE:
                  /* unless we have never held threads for the life */
                  /* of the process */
                  if (pthreaded)
                  {
                      /* debugger must handle list of any size */
                      struct pthread commit_tids;
                      int commit_count = 0;
                      /* debugger must handle list of any size */
                      struct pthread continue_tids;
                      int continue_count = 0;
		      
		      rc = pthdb_session_committed(session, committed_symbol);
		      if (rc != PTHDB_SUCCESS)
			  /* handle error */
	              /* set break point  at committed_symbol */	
		      
                      /* gather any tids necessary to commit hold/unhold */
                      /* operations */
                      do
                      {
                          rc = pthdb_session_commit_tid(session, 
                                                &commit_tids.th[commit_count++]);
                          if (rc != PTHDB_SUCCESS)
                              /* handle error and exit */
                      } while (commit_tids.th[commit_count - 1] != PTHDB_INVALID_TID);
  
                      /* set up thread which stopped the process to be */
                      /* parked using the stop_park function*/
  
		      if (commit_count > 0) {
                        rc = ptrace(PTT_CONTINUE, stop_tid, stop_park, 0, 
                                                              &commit_tids);
                        if (rc)
                            /* handle error and exit */
  
                        /* wait on process to stop */
		      }
  
                      /* gather any tids necessary to continue */
                      /* interesting threads */
                      do
                      {
                          rc = pthdb_session_continue_tid(session, 
                                          &continue_tids.th[continue_count++]);
                           if (rc != PTHDB_SUCCESS)
                              /* handle error and exit */
                      } while (continue_tids.th[continue_count - 1] != PTHDB_INVALID_TID);
  
                      /* add interesting threads to continue_tids */
  
                      /* set up thread which stopped the process to be parked */
                      /* unless it is an interesting thread */
  
                      rc = ptrace(PTT_CONTINUE, stop_tid, stop_park, 0, 
                                                                &continue_tids);
                      if (rc)
                          /* handle error and exit */
                  }
              case DB_EXIT:
		rc = pthdb_session_destroy(session);
		/* other clean up code */
		exit(0);
              ...
          }
      }
  
    }
    exit(0);
}

Related Information

The pthread.h file.


[ Previous | Next | Table of Contents | Index | Library Home | Legal | Search ]