Allows insertion of instrumentation code at the entry and exit points of existing program and library subroutines.
stem { -mf MapFile | -p Program | -pshm Program } [ -ld_cmd ] [ -ld_options QuotedString ] [ -noleaf ][ -libdir Directory ] [ -exedir Directory ]
stem { -on [ -noreset ] | -off | -shm [Size ] | -shmkill }
The stem (Scanning Tunneling Encapsulating Microscope) command is a tool for inserting instrumentation code, either user-supplied or default routines provided with stem, in subroutines. The stem command operates on existing libraries and programs without requiring source code or recompilation of the libraries or programs.
The stem command places instrumentation at the entry and exit points of selected subroutines, known as target routines. Target routines can be in user programs or shared libraries. User-defined instrumentation is in the form of subroutines, known as instrumentation routines. Instrumentation routines are simple C subroutines readily created and tailored by you.
In addition to instrumenting target routines, the stem command can replace target routines with instrumentation routines.
Note: The stem command does not require an installed C compiler.
Note: This flag pins the shared-memory buffer; be careful not to make the buffer too large.
Though stem instrumentation routines can print to files and call the trace daemon, another output mechanism is a specially constructed shared-memory segment. The shared-memory segment enables shared-library and user-program target routines to share one output stream.
The current buffer design includes four sections, as shown in the "stem Shared-Memory Layout" illustration.
The /usr/lpp/stem/headers/stem_shm.h header file defines the buffer structure and event types.
The Control Area includes the ON/OFF flag, pointers defining the boundaries of the other areas, and the WithinInstrumentation flag. WithInstrumentation determines the present instrumentation state and prevents infinite instrumentation loops.
Infinite instrumentation loops result from instrumenting a target routine and then directly or indirectly calling the target routine from the instrumentation routine. Assume the following Stem_instr_routine():
Stem_instr_routine() {
printf("Call results in an infinite instrumentation
loop\n");
printf("if the printf routine is instrumented\n);
}
instruments the target routine printf().
The instrumented program encounters an infinite instrumentation loop because calls to printf() are directed to Stem_instr_routine() and printf() is directly called by Stem_instr_routine(). The WithinInstrumentation flag in the shared-memory buffer and a test of the flag in each instrumentation routine prevents infinite instrumentation loops. Properly coded, Stem_instr_routine() appears as follows:
Stem_instr_routine() {
if ( shmbuf[ WithinInstrumentation ] ) return();
shmbuf[ WithinInstrumentation ] = 1; /* turn on flag */
printf("not an infinite instrumentation loop\n");
shmbuf[ WithinInstrumentation ] = 0; /* turn off flag */
}
Of course the preceding segment assumes addressability to the shared-memory buffer has already been established by global variable shmbuf, an integer pointer. See the file /usr/samples/perfagent/stem/stem_samples.c for example instrumentation routines and usage of the WithinInstrumentation flag.
Note: There exists only one WithinInstrumentation flag within the shared-memory buffer. You require multiple flags if more than one stem instrumented program runs at the same time.
The Counter Area is reserved for a counter-based implementation. For example, rather than producing an event for each file open, instrumentation routines can add to a counter or set of counters.
The PID to Process Name Map Area contains the process IDs and process names for all processes running on the system at the time of the last stem -on call.
The Event Log Area houses events of any type and any length. The stem command defines some event types in the /usr/samples/perfagent/stem/stem_shm.h header file. Each event has the structure shown in the stem Event Structure illustration .
The first word of the event stores the type and length markers. The length represents the number of 32-bit words, including the first word.
Both instrumented and uninstrumented programs can make use of instrumented libraries by setting the LIBPATH environment variable. The stem command prints a message showing what setting to use for LIBPATH whenever a shared library is instrumented. Unless the -libdir flag specifies another value, the LIBPATH environment variable should be set to /tmp/LIBS. The following command makes use of an instrumented library located in /tmp/LIBS:
LIBPATH=/tmp/LIBS /bin/ps -eaf
The preceding uninstrumented /bin/ps program will use whatever instrumented libraries exist in the directory /tmp/LIBS.
The stem command creates the ./stemdir subdirectory in the current working directory. Under the ./stemdir directory, stem creates additional subdirectories, one directory for each executable or library to instrument. For the following map file,
StemAll /bin/some_program .Stem_ShmEnter .Stem_ShmExit stem_samples.o StemAll /usr/lib/libc.a:shr.o .Stem_ShmEnter .Stem_ShmExit stem_samples.o .main runit .Stem_ShmEnter No_Exit stem_samples.o
the stem command creates the following three directories:
Notice that each slash (/) in the second column of the map file has been substituted with an underline character (_). Two files in each stem-created directory are of particular interest: instrumented and not_instrumented. The instrumented file contains the list of target routines instrumented by stem. The not_instrumented file contains the list of target routines not instrumented by stem and a brief reason why they were not instrumented.
The stem command also creates directories to store instrumented libraries and instrumented programs. The stem command places:
A program is stripped if the strip command removed its symbol table. The stem command can instrument both stripped and un-stripped programs, but it only instruments un-stripped libraries.
The instrumentation method for stripped and un-stripped programs are different. For un-stripped programs, stem uses the ld command, the linkage editor, to combine the program and the instrumentation object file. The ld command is responsible for resolving external references and searching libraries. The stem command tries to provide the necessary parameters to ld but sometimes is unable to do so. Such cases require user intervention.
The stem command flags -ld_options and -ld_cmd assist user intervention. The -ld_options flag enables you to pass ld the necessary options required to resolve external references. The -ld_cmd flag is similar though more direct. The stem command creates a shell procedure, named ld_cmd, in the stem-created directory for the program. You can change this shell procedure, and then use stem with the -ld_cmd flag to run it.
The stem command instruments stripped programs in an entirely different manner because the ld command does not operate on stripped programs. Therefore, the instrumentation object file cannot be combined with the stripped program. To instrument stripped programs, stem creates a dependency between the stripped program and a library containing the instrumentation object code. This is all done automatically by stem.
Note: The ld command is not used by stem on stripped programs. If ld problems continue when instrumenting an un-stripped program, try stripping the program before having stem instrument it.
Instrumentable target routines represent most of the routines within programs and libraries. However, stem cannot instrument certain assembler language routines and a few compiler generated routines. Before instrumenting a target routine, stem carefully evaluates the sequence of instructions within the routine. If the routine does not appear to be following normal linkage conventions, stem will not instrument it.
The stem command uses the stripnm program to locate the addresses of routines within programs and libraries. Though uncommon but not impossible, if stripnm cannot find a routine, stem will not be able to instrument it.
The stem command operates by inserting instrumentation routines in existing programs and libraries, and by changing the program flow. The instrumentation routines can perform most operations but there are limitations. Instrumentation routines are being inserted within the framework of an existing program or library and must conform to the execution context of that environment, which can be restrictive.
Note: The stem command does not work on non-archived libraries and does not support programs or libraries with more than 6200 subroutines.
For example, instrumentation routines can typically open files and write to them without incident. However, C++ programs with exception handling are not supported. Also, some processes, aixterm for example, occasionally close all open files. This presents a problem to the instrumentation routines. They must detect the fact that one of their open files has been closed before performing any input or output operations on the file. Programming to detect this behavior is in the standard instrumentation routines in the /usr/samples/perfagent/stem/stem_samples.c file.
The stem command has been tested on a number of programming languages, C, C++, and Fortran, though C has received most of the testing. Most C and C++ programs and libraries can be instrumented without incident. Fortran programs, however, do pose one problem. The Fortran programming language supports multiple entry points to subroutines. The stem command cannot currently detect multiple entry points. Therefore, the stem output cannot properly represent entry and exit events from some Fortran routines.
The stem command does not work on programs containing LISP routines and cannot detect if a program was built with LISP routines in it.
The stem command does not modify the parameters to or return values from target routines. Instrumentation routines can, however, access these without exception. The only difficulty is discerning the types of these parameters and return values. For example, consider the following simple instrumentation routine, Stem_simple_entry().
Stem_simple_entry(a, b, c)
unsigned a;
long b;
int c;
{
printf(" %u %x %d\n", a, b, c);
}
With stem, it is possible to send hundreds or thousands of target routines to this instrumentation routine, most of which are not likely to have three parameters. Fewer still will have three parameters whose types match those in the preceding example. This does not pose a problem for the preceeding instrumentation code. The stem command passes parameters to routines in registers 3 through 10. In the preceding example, register 3's contents is placed in parameter a, even if a's type does not match the type being passed in to the target routine. Parameter b, contains the contents of register 4, and c has the contents of register 5. So, even if a routine has only one parameter, b and c will still be assigned the contents of registers 4 and 5 respectively.
Conventions mandate that return values from target routines be placed within register 3. Therefore, in the preceding routine, a would contain the return value from any routines that used Stem_simple_entry() as their exit point instrumentation routine.
CAUTION: The preceding instrumentation routine does not make use of any pointers to structures or pointers to character strings. Programs can encounter a segmentation error if an instrumentation routine attempts to reference with a parameter when the parameter was not a valid pointer. Consider the following:
Stem_dangerous_entry(a, b, c)
char *a;
long b;
int c;
{
printf(" %s %x %d\n", a, b, c);
}
Stem_dangerous_entry() is dangerous because the instrumentation routine tries to print a string pointed to by a. The preceding only works if the first parameter to target routines instrumented with Stem_dangerous_entry() is always a pointer to a character string.
Access Control: You must be root or a member of the perf group to run this command.
To get started, simply choose a program to instrument and run the following commands:
stem -p Program
/tmp/EXE/Program
/bin/pg stem_out_*
The -p flag instructs stem to instrument all instrumentable target routines of Program with the routines Stem_Standard_entry() and Stem_Standard_exit(). The "Instrumentable Target Routines" section of this article contains a complete definition of instrumentable.
By default, the stem command puts the instrumented executable in the /tmp/EXE/Program file and leaves the original Program file unaltered. Include the full path after the -p flag if Program does not reside in the current directory.
Instrumentation routines Stem_Standard_entry() and Stem_Standard_exit() produce one line of output each time a routine is entered or exited as /tmp/EXE/Program runs. The stem command pairs and indents the routine entry and exit events to reflect the nesting level within the program.
The stem output from running /tmp/EXE/Program is directed to one or more output files, (stem_out_001, stem_out_002, stem_out_003,..., stem_out_nnn). The numeric suffix _nnn, designates the execution context of the instrumented program. An execution context can be signal handling code, or different threads. Many instrumented programs have just one execution context and only create the file stem_out_001. This article refers to execution contexts and numeric suffixes as threads and thread IDs (TID), respectively.
If Program in the preceding were /bin/ps, output in stem_out_001 would appear as:
Seconds.usecs TID Routine Names & Seconds.usecs since entering routine. 766011575.869091 1 -> main 766011575.906776 1 -> setlocale 766011575.907578 1 <-setlocale 0.000802 766011575.907741 1 -> catopen 766011575.907881 1 <-catopen 0.000140 766011575.908019 1 -> setbuf 766011575.908217 1 <-setbuf 0.000198 766011575.908366 1 -> parseberkeley 766011575.908497 1 <-parseberkeley 0.000131 766011575.908636 1 -> getpagesize 766011575.908861 1 <-getpagesize 0.000225 766011575.909093 1 -> ttyname 766011575.909698 1 <-ttyname 0.000605 766011575.909873 1 -> strncmp 766011575.910186 1 <-strncmp 0.000313 ..... 766011576.642508 1 -> prcom 766011576.642635 1 -> getthreaddata 766011576.642766 1 -> getthrds 766011576.642927 1 <-getthrds 0.000161 766011576.643067 1 -> malloc 766011576.643206 1 <-malloc 0.000139 766011576.643350 1 -> getthrds 766011576.643513 1 <-getthrds 0.000163 766011576.643654 1 <-getthreaddata 0.001019 766011576.643794 1 -> getprocftdata 766011576.643924 1 -> malloc 766011576.644061 1 <-malloc 0.000137 766011576.644204 1 <-getprocftdata 0.000410 766011576.644344 1 -> gettty 766011576.644475 1 <-gettty 0.000131 766011576.644616 1 <-prcom 0.002108 766011576.644752 1 -> fclose 766011576.644910 1 <-fclose 0.000158 766011576.645051 1 -> done 766011576.645180 1 -> exit
The first column contains the time (seconds and microseconds) of the enter or exit event. The second column has the stem thread ID (TID), which always corresponds to the suffix of the particular stem_out_nnn file. The first and second columns appear as they do so a sorted merge of all stem_out_* files will produce an easy-to-read stream of output, clearly showing the order of routine entries and exits across thread IDs.
The indented routine name, prefixed with either the routine entry symbol (-> ) or the routine exit symbol (<- ), appears to the right of the TID column. The indentation is meant to reflect the calling sequence or callgraph of the instrumented program. For exit events, one additional column appears and includes the elapsed time (seconds and microseconds) since entering the routine.
The preceding sample output contains one additional noteworthy characteristic. The routine named exit() has an enter event but no corresponding exit event. In fact, no additional enter or exit events appear in the output. The Stem_Standard_entry() instrumentation routine specifically recognizes the exit() routine because exit() begins the process of destroying the data structures and files used by the Stem_Standard_entry() and Stem_Standard_exit(). Upon encountering exit(), Stem_Standard_entry() sets a flag to avoid logging data for all subsequent enter and exit events.
The instrumentation source file, /usr/samples/perfagent/stem/stem_samples.c, shows the Stem_Standard_entry(), and Stem_Standard_exit() routines. You can easily modify these routines. The -p flag copies the stem_samples.c file to the current directory unless a stem_samples.c file exists in the current directory.
The -pshm flag is similar to the -p flag. The difference is in the instrumentation routines and the output mechanisms. The -pshm flag uses the Stem_ShmEnter() and Stem_ShmExit() routines to instrument each target routine within Program.
Stem_ShmEnter() and Stem_ShmExit() do not open files to log output, they log output in a specially-made shared-memory buffer. There are a few advantages to this approach.
A disadvantage is post processing. You must run stem again using the -cg flag to view the data. The following is an example sequence of instructions using the -pshm and -cg flags:
stem -pshm /bin/ps stem -on /tmp/EXE/ps stem -cg /tmp/EXE/ps PID ElapsedTime DeltaSecs IAR NAME 9500 Enter 0.000000 0.000000 10000178 1 main 9500 Enter 0.000290 0.000290 1000a7d0 1 . setlocale 9500 Exit 0.001086 0.000796 1000a7d0 1 . setlocale 9500 Enter 0.001120 0.000034 1000a7f4 1 . catopen 9500 Exit 0.001223 0.000103 1000a7f4 1 . catopen 9500 Enter 0.001249 0.000026 1000a818 1 . setbuf 9500 Exit 0.001388 0.000139 1000a818 1 . setbuf 9500 Enter 0.001422 0.000034 10008ad0 1 . parseberkeley 9500 Exit 0.001448 0.000026 10008ad0 1 . parseberkeley 9500 Enter 0.001474 0.000026 1000afd4 1 . getpagesize 9500 Exit 0.001591 0.000117 1000afd4 1 . getpagesize 9500 Enter 0.001673 0.000082 1000b040 1 . ttyname 9500 Exit 0.002156 0.000483 1000b040 1 . ttyname 9500 Enter 0.002193 0.000037 1000a8cc 1 . strncmp 9500 Exit 0.002297 0.000104 1000a8cc 1 . strncmp
The instrumentation source file, /usr/samples/perfagent/stem/stem_samples.c shows the Stem_ShmEnter(), and Stem_ShmExit() routines. You can easily modify these routines. The -pshm flag copies the stem_samples.c file to the current directory unless a stem_samples.c file exists in the current directory.
Use the -p and -pshm flags to instrument all instrumentable target routines within a specified program with the standard instrumentation routines. Use the -mf flag to instrument library target routines or to better control the instrumentation of program target routines.
The Map_File parameter specifies a file you create whose contents describe which target routines in user programs and/or shared libraries to instrument. Map files contain the names and locations of target routines and of the entry and exit instrumentation routines. The following example shows the format of a stem map file:
Target Target Entry Exit Instrumentation
Routine Routine Instrumentation Instrumentation Object File
File Routine Routine
The following entry is an example of a stem map file written as one line.
.malloc /usr/lib/libc.a:shr.o .Stem_malloc_entry .Stem_malloc_exit stem_samples.o
Specify target and instrumentation routine names with a . (dot) as the first character. This is how they are reported by the /usr/bin/nm and /usr/bin/stripnm programs.
To avoid name collisions with routines in the target files, stem mandates that all instrumentation routines begin with the "Stem_" prefix. In fact, all routines in an instrumentation object file must begin with the "Stem_" prefix. Instrumentation object files in column 5 can have any name, but they should reside within the current directory.
The preceding map file changes the program flow of target routine malloc(). After instrumentation, calls to target routine malloc() are directed first through instrumentation routine Stem_malloc_entry(), then through malloc(), and finally through instrumentation routine Stem_malloc_exit(). The calling sequence becomes:
Stem_malloc_entry() ---> malloc() ---> Stem_malloc_exit()
Target routine malloc() must exist within the shared library target file /usr/lib/libc.a. Because the library archive /usr/lib/libc.a contains several object files, such as shr.o and meth.o, the stem map file must identify the correct object because target routine malloc() could exist in more than one of the archive's object files. A colon (:) separates the object file from the archive name.
Instrumentation routines Stem_malloc_entry() and Stem_malloc_exit() must exist in instrumentation object file stem_samples.o. Also, stem_samples.o must reside in the current directory. If the file Make.Stem exists in the current directory, stem tries to create the instrumentation object file by running the command:
make -f Make.Stem stem_samples.o.
If the file Make.Stem does not exist, stem tries to create the instrumentation object file by running the command:
make stem_samples.o.
Note: The stem software includes a sample instrumentation source file, stem_samples.c, in the directory /usr/samples/perfagent/stem. It contains several example instrumentation routines to use as is, or they can be modified to suit your specific requirements.
To instrument all routines in a program or shared library, list each routine separately in the map file, or use the keyword StemAll. In the following map file example, all instrumentable target routines in /bin/some_program and /usr/lib/libc.a:shr.o are directed through the instrumentation routines Stem_ShmEnter() and Stem_ShmExit().
Note: Instrumentable target routines represent most routines. The stem command cannot instrument certain assembler-language routines. The stem command creates an "instrumented" and not a "not_instrumented" file for each library or user program specified in column 2 of the map file. See the "Instrumentable Target Routines" section of this article for additional details.
StemAll /bin/some_program .Stem_ShmEnter .Stem_ShmExit stem_samples.o StemAll /usr/lib/libc.a:shr.o .Stem_ShmEnter .Stem_ShmExit stem_samples.o
To instrument only the entry point of a target routine, use the keyword No_Exit in column 4, the exit instrumentation routine column. In the following example map file, all routines in /bin/some_program have their entry points instrumented but not their exit points.
StemAll /bin/some_program .Stem_ShmEnter No_Exit stem_samples.o
To replace a target routine with an instrumentation routine, use the keyword Replace in column 4, the exit instrumentation column. In the following example map file, all calls to the some_targ() routine in /bin/some_program will be directed to the instrumentation routine Stem_replace_example().
.some_targ /bin/some_program .Stem_replace_example Replace stem_samples.o
Note: Target routine replacement is inherently dangerous. To avoid unpredictable results, the replacement routines must adhere to all pre-conditions and post-conditions of the target routines. For example, you can replace a sorting routine with another as long as the output is properly sorted upon exiting the replacement routine.
The stem command uses a shared-memory segment to control program flow, to assist communication between user programs and libraries, and to store data in the form of events. The stem command requires the existence of the shared-memory buffer and will create one even if no shared-memory flags are specified. The "Shared-Memory Buffer" section describes the layout of the shared-memory buffer.
Multiple postprocessing programs can process the shared-memory buffer at the same time. All stem-instrumented user programs and shared libraries can write to the shared-memory buffer. Use the-shm flag to specify the size of the buffer. You can increase the buffer size but not reduce it. The -shm flag without the Size parameter displays the current size of the shared-memory.
Note: The -shmkill flag can destroy the buffer.
The flags -on, -off, and noreset, control the logging of events. Logging of events only occurs if the shared-memory buffer's ON/OFF flag is set to ON and the buffer is not full. When created, the buffer's current pointer is set to full and the ON/OFF flag is OFF. The command:
stem -on
resets the buffer pointer and turns on the ON/OFF flag. This command also stores the process IDs and process names of all running processes in the shared-memory buffer.
CAUTION: Since the commands stem and syscalls share the same buffer, do not run them at the same time.
The ld command, strip command, stripnm command, and, syscalls command.