IBM Books

Diagnosis Guide


Trace information

ATTENTION - READ THIS FIRST

Do not activate this trace facility until you have read this section completely, and understand this material. If you are not certain how to properly use this facility, or if you are not under the guidance of IBM Service, do not activate this facility.

Activating this facility may result in degraded performance of your system. Activating this facility may also result in longer response times, higher processor loads, and the consumption of system disk resources. Activating this facility may also obscure or modify the symptoms of timing-related problems.

The log files, including the Group Services Trace logs and startup logs, are preserved as long as their total size does not exceed 5MB. If the total size is greater than 5MB, the oldest log file is removed at Group Services startup time.

GS service log trace

The GS service log contains a trace of the GS daemon. It is intended for IBM Support Center use only, and written in English. It refers to a particular instance of the GS daemon running on the local node. When a problem occurs, logs from multiple nodes are often needed.

If obtaining logs from all nodes is not feasible, collect logs from these nodes:

Service log short tracing is always in effect. Service log long tracing is activated by this command:

traceson -l -s subsystem_name

where subsystem_name is:

The trace is deactivated, (reverts to short tracing) by issuing this command:

tracesoff -s subsystem_name

where subsystem_name is the same as for trace activation.

The trace may produce 20MB or more of data, depending on GS activity level and length of time that the trace is running. Ensure that there is adequate space in the directory /var/ha.

The trace is located in:

where incarnation is an increasing integer set by the GS daemon. This value can be obtained from the NodeId field of the command:
hagsns -l -s gssubsys

The long trace contains this information:

  1. Each Group Services protocol message sent or received
  2. Each significant processing action as it is started or finished
  3. Details of protocols being run

For many of the cases, log files from multiple nodes must be collected. The other nodes' log files must be collected before they wrap or are removed. By default, during the long tracing, log files will expand to a maximum of 5 times the configured log size value.

To change the configured value of the log size on a PSSP node, issue this command:

SDRChangeAttrValues TS_Config Log_Length=new_length 

where new_length is the number of lines in the trace log file. Then, restart the GS daemon.

To change the configured value on a HACMP node, perform these steps:

  1. Issue this command: smit hacmp.
  2. Select Cluster Configuration.
  3. Select Cluster Topology.
  4. Select Configure Topology Services and Group Services.
  5. Select Change/Show Topology and Group Services Configuration.
  6. Select Group Services log length (number of lines).
  7. Enter the number of lines for each Group Services log file.

When the log file reaches the line number limit, the current log is saved into a file with a suffix of .bak. The original file is then truncated. With the "long" trace option, the default of 5000 lines should be enough for only 30 minutes or less of tracing.

Each time the daemon is restarted, a new log file is created. Only the last 5 log files are kept.

Long tracing should be activated on request from IBM Service. It can be activated (for about one minute, to avoid overwriting other data in the log file) when the error condition is still present.

Each entry is in the format: date message.

The "short" form of the service log trace is always running. It contains this information:

  1. Each Group Services protocol message sent or received.
  2. Brief information for significant protocols being run.
  3. Significant information for possible debugging.

GS service log trace - summary log

The GS service log - summary log contains a trace of the GS daemon, but records only important highlights of daemon activity. This log does not record as much information as the GS service log, and therefore it will not wrap as quickly as the GS service log. This log is more useful in diagnosing problems whose origin occurred a while ago. All information in this log is also recorded in the GS service log, provided that the log has not yet wrapped. The GS service log - summary log is intended for IBM Support Center use only, and written in English. It refers to a particular instance of the GS daemon running on the local node. When a problem occurs, both logs from multiple nodes are often needed.

The trace is located in:

where incarnation is an increasing integer set by the GS daemon. This value can be obtained from the NodeId field of the command:
hagsns -l -s gssubsys

All other information about this log is identical to the GS service log. See GS service log trace.

Group Services startup script log

This log contains the GS daemon's environment variables and error messages where the startup script cannot start the daemon. The trace refers to a particular instance of the GS startup script running on the local node. This trace is always running. One file is created each time the startup script runs. The size of the file varies from 5KB to 10KB.

On PSSP nodes, it is located in: /var/ha/log/hags.default.partition_name.node_incarnation.

On HACMP nodes, it is located in: /var/ha/log/grpsvcs.default.domain.node_incarnation.

The main source for diagnostic information is the AIX error log. The GS startup script log should be used when the error log shows that the startup script was unable to complete its tasks and could not start the daemon.

The data in this file is in English. This information is for use by the IBM Support Center. The format of the data is the same as that of the GS Service Log Trace, "long" option.


[ Top of Page | Previous Page | Next Page | Table of Contents | Index ]