Diagnosis Guide

Information to collect before contacting the IBM Support Center

The following information needs to be collected from the node that presents the problem. For connectivity-related problems, the same information is needed from the other nodes. If collecting data from all the nodes is not feasible, data should be collected from at least the following nodes:

1. The node's Downstream Neighbor on all networks. This is the node whose IP address is immediately lower than the address of the node where the problem was seen. The node with the lowest IP address has a Downstream Neighbor of the node with the highest IP address.
2. The Group Leader node, which is the node with the highest IP address in the network.
3. The control workstation

Collect the files listed in number 1, 3d, and 3c (bullets 2 and 3). Then, issue phoenix.snap to collect the remaining information. See phoenix.snap dump.

1. FFDC dump files: /var/adm/ffdc/dumps/*
2. Topology Services Files
   • Service and User log files for the daemon.
     • On PSSP, /var/ha/log/hats.dd.hhmmss.partition_name*.
     • On HACMP, /var/ha/log/topsvcs.dd.hhmmss.cluster_name*.

     Here, dd is the day of the month when the daemon was started, and hhmmss is the time when the daemon was started.

   • Startup Script log.
     • On PSSP, /var/ha/log/hats.partition_name*.
     • On HACMP, /var/ha/log/topsvcs.default*.
   • Entire contents of the daemon run directory.
     • On PSSP, /var/ha/run/hats.partition_name/*.
     • On HACMP, /var/ha/run/topsvcs.cluster_name/*.

     This includes the machines.lst file and temporary files used by the startup script. It also includes the log files for the /usr/sbin/rsct/bin/netmonAdapterHealth program, which is used to determine adapter status. The daemon's core files are also stored here.

   • Output of the lssrc -ls command.
     • On PSSP nodes, issue the command: lssrc -ls hats.
     • On the PSSP control workstation, issue the command: lssrc -ls hats.partition_name.
     • On HACMP, issue the command: lssrc -ls topsvcs.
   • Information used by the startup scripts:
     • On any PSSP node, issue these commands and record the output:
       1. SDRGetObjects -x SP cw_ipaddrs
       2. SDRGetObjects -x TS_Config Frequency Sensitivity Run_FixPri FixPri_Value Log_Length Pinning
       3. SDRGetObjects -x Subnet
       4. SDRGetObjects -x Network
       5. SDRGetObjects -x Adapter
       6. SDRGetObjects -G -x Adapter
       7. SDRRetrieveFile hats.machines.lst local_file_name
       8. SDRGetObjects host_responds
       9. /usr/lpp/ssp/bin/lsauthpts -c
       10. /usr/bin/dcecp -c keytab show rsct/partition_name/hats -member
       11. /usr/lpp/ssp/bin/splstdata -p
     • On HACMP nodes, issue these commands and record the output:
       1. odmget HACMPnim
       2. odmget HACMPtopsvcs
       3. /usr/es/sbin/cluster/utilities/clhandle -ac
       4. /usr/es/sbin/cluster/utilities/clhandle -c
       5. /usr/es/sbin/cluster/utilities/cllsif -Sc
       6. /usr/es/sbin/cluster/utilities/cllsclstr -Sc
       7. /usr/es/sbin/cluster/utilities/clmixver (output and return value)
3. System Data
   1. Installation data
      • Output of the command: lslpp -L
      • Contents of files:
        • /usr/sbin/rsct/optlevel.rsct.basic.rte
        • /usr/sbin/rsct/optlevel.rsct.basic.hacmp
        • /usr/sbin/rsct/optlevel.rsct.basic.sp
   2. The authentication method in use. Issue this command on the control workstation:

      splstdata -p
      

      The entry "ts_auth_methods" lists the active authentication methods in use for the SP trusted services (such as Topology Services).

   3. Network and adapter data
      • Output of commands:
        1. netstat -in
        2. ifconfig (all interfaces)
        3. netstat -s
        4. netstat -m
        5. netstat -rn
        6. netstat -D
        7. netstat en* (for example, en0, en1 and so forth)
      • If the problem is still occurring, output of ping to addresses in the machines.lst file, located in the daemon run directory.
      • If the problem is still occurring, output of the iptrace command. Follow this sequence:
        1. iptrace /tmp/iptrace.out
        2. ps -ef | grep iptrace
        3. Wait one minute, and then issue: kill pid
        4. Save a copy of the file /tmp/iptrace.out. This is a binary file.
   4. Memory - output of the commands:
      1. vmstat 5 5
      2. vmstat -s
      3. vmtune

      The full path name is: /usr/samples/kernel.