Debug and Performance Tracing

[ Previous | Next | Contents | Home | Search ]

AIX Version 4.3 Kernel Extensions and Device Support Programming Concepts

Memory Overlay Detection System (MODS)

AIX Kernel Memory Overlay Detection System (MODS)

Some of the most difficult types of problems to debug are what are generally called "memory overlays." Memory overlays include the following:

Writing to memory that is owned by another program or routine
Writing past the end (or before the beginning) of declared variables or arrays
Writing past the end (or before the beginning) of dynamically-allocated memory
Writing to or reading from freed memory
Freeing memory twice
Calling memory allocation routines with incorrect parameters or under incorrect conditions.

In the kernel environment (including the AIX kernel, kernel extensions, and device drivers), memory overlay problems have been especially difficult to debug because tools for finding them have not been available. Starting with AIX Version 4.2.1, however, the Memory Overlay Detection System (MODS) helps detect memory overlay problems in the AIX kernel, kernel extensions, and device drivers.

Note: This feature does not detect problems in application code; it onl watches kernel and kernel extension code.

bosdebug command
The bosdebug command turns the MODS facility on and off. Only the root user can run the bosdebug command.

To turn on the base MODS support, type:

                                                                            
bosdebug -M

For a description of all the available options, type:

                                                                            
bosdebug -?

Once you have run bosdebug with the options you want, run the bosboot -a command, then shutdown and reboot your system (using the shutdown -r command). If you need to make any changes to your bosdebug settings, you must run bosboot -a and shutdown -r again.

When to use the MODS feature
This feature is useful in the following circumstances:

When developing your own kernel extensions or device drivers and want to test them thoroughly. third-party developers and want to further isolate the problems.
When asked to turn this feature on by IBM AIX service to help in further diagnosing a problem that you are experiencing.

How MODS works
The primary goal of the MODS feature is to produce a dump file that accurately identifies the problem.

MODS works by turning on additional checking to help detect the conditions listed above. When any of these conditions is detected, your system crashes immediately and produces a dump file that points directly at the offending code. (Previously, a system dump might point to unrelated code that happened to be running later when the invalid situation was finally detected.)

If your system crashes while the MODS is turned on, then MODS has most likely done its job.

To make it easier to detect that this situation has occurred, the crash command has been extensively modified. The stat subcommand of crash now displays both:

Whether the MODS (also called "xmalloc debug") has been turned on
Whether this crash was the result of the MODS detecting an incorrect situation.

The xmalloc subcommand provides details on exactly what memory address (if any) was involved in the situation, and displays mini-tracebacks for the allocation and/or free of this memory.

Similarly, the netm command displays allocation and free records for memory allocated using the net_malloc kernel service (for example, mbufs, mclusters, etc.).

You can use these commands, as well as standard crash techniques, to determine exactly what went wrong.

MODS limitations
There are limitations to the Memory Overlay Detection System. Although it significantly improves your chances, MODS cannot detect all memory overlays. Also, turning MODS on has a small negative impact on overall system performance and causes somewhat more memory to be used in the kernel and the network memory heaps. If your system is running at full CPU utilization, or if you are already near the maximums for kernel memory usage, turning on the MODS may cause performance degradation and/or system hangs.

Our practical experience with the MODS, however, is that the great majority of customers will be able to use it with minimal impact to their systems.

MODS benefits
You'll see these benefits from using the MODS:

You can more easily test and debug your own kernel extensions and device drivers.
Difficult problems that once required multiple attempts to recreate and debug them will generally require many fewer such attempts.

[ Previous | Next | Contents | Home | Search ]