Diagnosis Guide

Dump information

Group Services creates a core dump automatically when certain errors occur, and also provides service information that can be obtained automatically by the phoenix.snap script.

Core dump

A core dump is generated by the Group Services daemon if it encounters an undefined condition. It contains normal information saved by AIX in a core dump: Group Services daemon's process user data and call stack information. The dump is specific to a particular instance of the GS daemon on the local node. Other nodes may have a similar core dump. Each core dump file is approximately 10MB in size.

For a PSSP node, the core dumps are located in: /var/ha/run/hags.partition/core* and /var/ha/run/hagsglsm.partition/core*.

For a HACMP node, the core dumps are located in: /var/ha/run/grpsvcs.cluster/core* and /var/ha/run/grpglsm.cluster/core*.

Core dumps are created automatically when:

• One of the GS daemons invokes an assert() statement if the daemon state is undefined or encounters an undefined condition by design.
• The daemon attempts an incorrect operation, such as division by zero.
• The daemon receives a segmentation violation signal for accessing its data incorrectly.

A core dump is created manually by issuing the command:

kill -6 pid_of_daemon

where pid_of_daemon is obtained by issuing the command:

lssrc -s gs_subsystem

where gs_subsystem is:

• hags on PSSP nodes
• hags.partition on the PSSP control workstation
• grpsvcs on HACMP nodes

The core dump is valid as long as the executable file /usr/sbin/rsct/bin/hagsd is not replaced. Copy the core dumps and the executable file to a safe place.

To verify the core dump, issue this command:

dbx /usr/sbin/rsct/bin/hagsd core_file

where core_file is one of the core* files described previously.

Good results are indicated by output similar to:

Type 'help' for help.
 reading symbolic information ... 
[using memory image in core]  
IOT/Abort trap in evt._pthread_ksleep [/usr/lib/libpthreads.a] 
at 0xd02323e0 ($t6) 0xd02323e0 (_pthread_ksleep+0x9c) 80410014
 lwz    r2,0x14(r1) 
 

Error results may look like one of the following:

1. This means that the current executable file was not the one that created the core dump.

   Type 'help' for help.
   Core file program (hagsd) does not match current program (core ignored)
   reading symbolic information ...
   (dbx)
    
   

2. This means that the dump is incomplete due to lack of disk space.

   Type 'help' for help.
   warning: The core file is truncated.  You may need to increase the ulimit
   for file and coredump, or free some space on the filesystem.
   reading symbolic information ...
   [using memory image in core]
    
   IOT/Abort trap in evt._pthread_ksleep [/usr/lib/libpthreads.a] 
   at 0xd02323e0
   0xd02323e0 (_pthread_ksleep+0x9c) 80410014
           lwz   r2,0x14(r1)
   (dbx)
   

phoenix.snap dump

This dump contains diagnostic data used for RSCT problem determination. It is a collection of log files and the other trace information for the RSCT components. For more detailed information about phoenix.snap, see the Topology Services chapter, heading phoenix.snap dump.