Diagnosis Guide

Error symptoms, responses, and recoveries

Use the following table to diagnose problems with the PSSP software for all servers.

Table 87. IBM pSeries 690 symptoms

Actions

Action 1 - Install ssp.basic file set, Java 1.3, RPM, and CIMOM

Perform Installation test 1 -Verify the ssp.basic file set to verify that this is a problem. Install the ssp.basic file set using the installp command. Repeat Installation test 1 -Verify the ssp.basic file set.

Perform Installation test 3 - Check external hardware daemon components to verify that this is a problem. Perform the following installation procedures. Repeat Installation test 3 - Check external hardware daemon components.

Java 1.3 Installation

  Java130.rte.bin            1.3.0.8    C     F  Java Runtime Environment
                                                 Executables
    - how to install
      smitty
    - where it gets installed
      /usr/java130/jre/bin/*

  Java130.rte.lib            1.3.0.8    C     F  Java Runtime Environment
    - how to install
      smitty
    - where it gets installed
      /usr/java130/jre/lib/*
                                                 Libraries
  Java130.xml4j              1.3.0.0    C     F  XML Parser for Java
    - how to install
      smitty
    - where it gets installed
      /usr/java130/jre/lib/ext/xerces.jar

RPM Installation

  rpm.rte                   3.0.5.30    C     F  RPM Package Manager
  AIX-rpm                   5.1.0.10    C     R  Virtual Package for libraries
                                                 and shells installed on system
                                                 (/bin/rpm)
    - how to install
      smitty
    - where it gets installed
      /bin/rpm

CIMOM Installation

  openCIMOM                     0.61    C     R  The SNIA CIMOM (Common
                                                 Information Model Object
                                                 Manager)  (/bin/rpm)
    - name of the CD that cimom is delivered on:
      "AIX toolbox for Linux applications"
    - file name on disk
      openCIMOM-0.61-1.aix5.1.noarch.rpm
    - how to install
      rpm -i openCIMOM-0.61-1.aix5.1.noarch.rpm
    - where it gets installed
      /opt/freeware/cimom/org/snia/wbem/client/*
      /opt/freeware/cimom/org/snia/wbem/cim/*
    - requires rpm.rte & AIX-rpm

Action 2 - Check permissions of ssp.basic

Perform Installation test 2 - Check external hardware daemon to verify that this is a problem. Install the ssp.basic file set using the installp command. If any of the permission or file attributes do not match what is shown in Installation test 2 - Check external hardware daemon, issue the chmod or chown commands, as appropriate, to correct the attributes.

Repeat Installation test 1 -Verify the ssp.basic file set and Installation test 2 - Check external hardware daemon.

Action 3 - Correct SDR Frame object

Perform Configuration test 1 - Check SDR Frame object to verify that this is a problem. Perform these steps:

1. If the SDRGetObjects Frame command that was run in Configuration test 1 - Check SDR Frame object returns an error message similar to:

   splstdata: 0022-001 The repository cannot be accessed. Return code was 80.
   

   refer to Diagnosing SDR problems to determine why the SDR cannot be accessed.

2. If you can successfully access the SDR, create a Frame object for the one that is missing by issuing the spframe command with the appropriate parameters. For a description of the spframe command, refer to PSSP: Command and Technical Reference.
3. If there were problems creating the SDR Frame object, investigate why the SDR_config command was unable to create the Frame object. Check the SDR_config log file, /var/adm/SPlogs/sdr/SDR_config.log for error messages.
4. If the Frame object was successfully created, but the server entry, output by the SDRGetObjects Frame command, does not have correct information in it's control_ipaddrs or domain_name fields, issue the spframe command with the correct options.
5. If the Frame object was successfully created, but the server entry, output by the SDRGetObjects Frame command, does not have correct information in it's hardware_protocol column, you must first issue the spdelfram command. Then, create a new definition by issuing the spframe command with the correct options.

For an IBM pSeries 690 type server, the value of the Frame object must be HMC. After correcting the problem, repeat Configuration test 1 - Check SDR Frame object.

Action 4 - Correct System Monitor polling interval

Perform Operational test 1 - Check hardmon status to verify that this is a problem. Perform these steps:

1. If the hardmon daemon is not running on the control workstation, you need to start it. Issue this command:

   startsrc -s hardmon
   

   Then, perform Operational test 1 - Check hardmon status again to determine if hardmon was started successfully.

2. If the hardmon daemon uses an incorrect polling interval, it may cause problems. The polling interval is chosen when the hardmon daemon is started by the AIX System Resource Controller. The value is in the cmdargs attribute in the hardmon ODM SRCsubsys object. Check the polling interval by issuing the ODM command:

   odmget -q subsysname=hardmon SRCsubsys
   

   The output is similar to the following, which is the default:

   SRCsubsys:
      subsysname = "hardmon"
      synonym = ""
      cmdargs = "-r 5"
      path = "/usr/lpp/ssp/bin/hardmon"
      uid = 0
      auditid = 0
      standin = "/dev/console"
      standout = "/dev/console"
      standerr = "/dev/console"
      action = 1
      multi = 0
      contact = 2
      svrkey = 0
      svrmtype = 0
      priority = 20
      signorm = 15
      sigforce = 15
      display = 1
      waittime = 15
      grpname = ""
   
   

   If the cmdargs attribute is not "-r 5", correct this by issuing the following command:

   chssys -s hardmon -a "-r 5"
   
   

   Then reissue the odmget command to verify that the new cmdargs attribute is "-r 5".After correcting the problem, repeat Operational test 1 - Check hardmon status.

Action 5 - Investigate external hardware daemon failure

Perform Operational test 2 - Check external hardware daemons to determine if the external hardware daemon is running. Perform Operational test 3 - Check frame responsiveness to determine if the external hardware daemon is responding. If either test produces error results, perform these steps:

1. Several components of PSSP, involved with the operation of the servers, write data to log files. Check these log files and take appropriate action:
   • The daemon log file. Refer to Daemon log.
   • The hardware monitor log. Refer to Hardware monitor log.
   • The SP hardware log. Refer to SP hardware log.
   • The AIX Error log.

   The same messages that are in the SP hardware log are also found in the AIX Error log. To obtain full details of all SP hardware messages in this log, issue the command:

   errpt -aN sphwlog
   
   

   You may want to redirect the output to a file, because there could be a large amount of output.

2. If one of the external hardware daemons is not running, but it should be, check to see if a core dump was created. Refer to Dump information.
3. If the System Monitor (hardmon) daemon is running, but an external hardware daemon is not running or not responding, issue the following command to start the external hardware daemon:

   hmcmds -G boot_supervisor F:0
   

   where F is the frame number of the server. This notifies the System Monitor that the external hardware daemon has stopped. The System Monitor then starts the daemon.

4. If you have attempted to start an external hardware daemon, and it still does not start, issue the following command to stop and restart the System Monitor daemon (hardmon):

   hmreinit
   

   The System Monitor daemon (hardmon) will be restarted by the AIX System Resource Controller, and the daemon will then restart all of the external hardware daemons. The SDR_config command will also run, updating the SDR as necessary.