Diagnosis Guide

Making effective use of the IBM Support Center

There are several things you need to know in order to make effective use of the IBM Support Center. You need to know when to call IBM, how to contact IBM, and what information to collect before calling.

When to contact the IBM Support Center

Contact the IBM Support Center for the following situations:

• A repeated or persistent halt of an SP node
• A repeated or persistent hang of an SP node
• A repeated or persistent failure or hang of specific SP software

  These failures may not always occur on the same node, given the distributed nature of this software.

• A failure in mission-critical PSSP software

A single node or infrequent software failure that is not mission-critical may not be a cause to contact the IBM Support Center immediately. These problems may be caused by conditions that can be remedied through administrative techniques. Investigate these failures, using this manual as a guide for conducting the investigation. Follow these steps:

• Determine what was active on the system at the time.
• See who was using the system.
• Record the date and time of the failure.
• Determine what hardware was in use.
• Determine what specific services were being used at the time that the failure was detected
• Use the information in this manual and Essential documentation - Other manuals to accompany this book to analyze and correct the problem.

Log information about these failures that you discover in the course of your investigations. This information can be used for your own future reference, and by the IBM Support Center if this failure becomes frequent enough or critical enough to require their assistance, as follows:

• The log information permits you to respond more quickly to similar failures in the future, and helps you to remember how to resolve the problem.
• The log information can be used for pattern analysis.

Problems and failures may appear to be unrelated at first, but they may have some relationship that is not immediately evident. Examine the conditions that were recorded for previous infrequent failures to see if there may be a pattern to them, even if the failure seem to be unrelated. Consider the following items when looking at the historical data on problems and failures:

• Do they happen when similar programs, procedures, or jobs are run?
• Do they happen when certain people or groups use the system?
• Do they happen at specific times, days, or shifts (peak or off-peak hours)?
• Does the failure occur when specific hardware is used?
• Are node reboots the only way to resolve the problem?

Contact the IBM Support Center when you discover any patterns in infrequent failures because:

• The system configuration may need repair.
• The IBM Support Center may have information about the problem you are experiencing.
• You may be experiencing a problem that no one else has ever encountered or reported.

Information to collect before contacting the IBM Support Center

1. Your Customer Information, which should be in the log discussed earlier. See Create a log of your SP structure and setup.
   • Your access code, which is your customer number
   • Names and phone numbers (external numbers, complete with area codes) where you can be reached by IBM service representatives
2. Your Product Information:
   • The PSSP product number for the level of PSSP running on the control workstation

     If PSSP 3.2 is running, the product number is 5765-D51. If a level of PSSP other than PSSP 3.2 is running, the product number can be obtained from PSSP: Diagnosis and Messages Guide for that level.

   • The PTF numbers for all PSSP fixes installed on the control workstation
   • The version number of AIX running on the control workstation
   • The PTF numbers for all AIX fixes installed on the control workstation
   • For all nodes involved in the problem, obtain this information:
     • The version number of AIX running on the node
     • The PTF numbers for all AIX fixes installed on the node
     • The PSSP version number on the node
     • The PTF numbers for all PSSP fixes installed on the control workstation
3. Information about your problem. Different information is needed for different kinds of problems. Therefore, you cannot collect the same set of information for all problems. Here are some general rules:
   • For a single node halt or crash, you will need the following information:
     1. A system dump from the halted node. The system may already have created this dump for you, but you must verify this. Examine the halted node's LED indicator using the spmon -Led command. If the display shows a flashing 888, a dump was started on the node. Use the spmon -reset command to step through the LED values until the flashing 888 appears again, recording all these LED values. Use Table 6 to determine if the dump has completed and verify its contents.
     2. The /unix file from the halted node. This file will be obtained automatically by the service tools.
     3. The error log from the halted node. This log will also be created by the service tools.
     4. System Data Repository (SDR) information from the control workstation, that is used to find environment or configuration problems.
        1. Login the control workstation as root.
        2. Issue the following commands, redirecting the output to a file:
           1. splstdata -e > filename
           2. splstdata -n -G >> filename
           3. splstdata -s -G >> filename
        3. Write the file to the media, and label it as "SDR information".

     To obtain this information:

     1. Verify the contents of the dump. Use Action 2. Verify the system dump.
     2. Ensure that the /tmp file system has at least 8MB of free space.
     3. Make sure that the /unix file is the one that was used when the dump occurred (in the case where the bosboot -k command was used to select another /unix).
     4. Login as root and issue the snap -r command to clear the current contents of the /tmp/ibmsupt directory.
     5. Make sure that a tape drive is accessible to the node.
     6. Issue the snap -Dgo tape device command.
     7. Label the media with the node name, node number, its contents (dump, /unix, snap information), and the command used to create it: snap -Dgo tape device.
     8. Enable write-protect on the media and put in a safe place.
     9. For additional information on this process, see RS/6000 SP: Problem Determination Guide .
   • For multiple node halts or crashes, you will need the same information as in single node halts and crashes. Examine each system using the spmon -Led command to determine if a system dump was taken on each halted node.

     To obtain the necessary information from the halted nodes:

     1. With the node still in its crashed or halted state, make sure that a system dump has been taken. If a dump does exist, do not re-create it. If a dump does not exist, one needs to be created. DO NOT create a dump if the node's LED shows flashing 888. Use Action 1. Produce a system dump to create the dump. Use the Primary Dump Device, unless this is impossible, because other tools will assume that the dump is located there.
     2. Verify the contents of the dump. Use Action 2. Verify the system dump.
     3. Once the node is rebooted and the dump verified, ensure that the /tmp file system has at least 8MB of free space available.
     4. Make sure that the /unix file is the one that was used when the dump occurred (in the case where the bosboot -k command was used to select another /unix).
     5. On the control workstation, ensure that the /tmp file system has at least 8MB of free space for each node that has halted. For example, if four nodes have halted, make sure that 32MB of space are available.
     6. On the control workstation, build a file containing the hostnames of the nodes that have halted or crashed. The hostnames should be on one line, separated by commas with no intervening white space characters. For example: node1a,node1b,node5d. Save the name of this file for use in the next two steps.
     7. On the control workstation, issue:

        splm -a service -t filename -r
        

        to clear the current contents of the /tmp/ibmsupt directories on these nodes. filename is the name of the file from Step f.

     8. On the control workstation, issue:

        splm -a service -t filename -c -p Dg
        

        to start the snap -Dg command on these nodes. filename is the name of the file from Step f.

     9. Ensure that a tape drive is available on the control workstation.
     10. On the control workstation, issue:

         splm -a gather -k service -t \
         filename -l /tmp/servcol -o tape_device_name
         

         to retrieve the service information. The command retrieves the information from the nodes listed in the file, writes this information temporarily to the /tmp/servcol file, then archives the data to the tape device in tar format. filename is the name of the file from Step f. tape_device_name is the name of the tape drive.

     11. If the /tmp/servcol file remains on the control workstation, remove it.
     12. Label the media with the names of the nodes involved, their node numbers, the contents of the tape (system dumps, /unix files, snap information), and the command used to create the tape from Step j.
     13. Enable the write protection on the media and put it in a safe place.
   • Node hangs or experiences response problems
     1. A dump from the hung node is preferred. Go through the steps given previously for manually creating and verifying a dump of the hung nodes.
     2. The /unix file is also needed, as in the previous procedure.
     3. Reboot the node and run the snap command given in the previous section to collect the data and create the media.
   • Failures in specific PSSP software subsystems, including denial of service problems and performance problems
     1. Consult Diagnosing PSSP subsystems and find the diagnosis chapter for the failing PSSP subsystem. This chapter may specifically request that you collect certain information for the node, including information from remote nodes that do not seem involved in the problem.
     2. If error log information is needed from multiple nodes, the splm command can be used to consolidate these logs in one location.
     3. Do not generate a system dump unless the subsystem's diagnosis chapter instructs you to do so.
     4. Write all information requested to the media and clearly label it.
   • Failure in other SP software which is supplied by IBM
     1. Consult the diagnosis documentation for the failing product. This information may specifically request that you collect certain information for the node, including information from remote nodes that do not seem involved in the problem
     2. Do not generate a system dump unless the product's diagnosis instructions instruct you to do so.
     3. Write all information requested to the media and clearly label it.
   • Failure in non-IBM software
     1. Consult the diagnostic documentation for the failing product. This information may specifically request that you collect certain information for the node, including information from remote nodes that do not seem involved in the problem.
     2. Follow problem reporting procedures for that product.
   • SP hardware failures
     1. Perform hardware diagnostic procedures associated with the hardware and record any information requested by these instructions. For details, see RS/6000 SP hardware publications.

How to contact the IBM Support Center

Contact the IBM Support Center using the phone number above, for these problems:

• Node halt or crash not related to a hardware failure
• Node hang or response problems
• Failure in specific PSSP software subsystems
• Failure in other SP software supplied by IBM

The person with whom you speak will ask for the information from Information to collect before contacting the IBM Support Center and give you a time period during which an IBM representative will return your call.

For failures in non-IBM software, follow the problem reporting procedures documented for that product.

For SP hardware failures, contact IBM Hardware Support at the number above.

For PC problems, contact the PC Help Center at the number above. Have your machine type and serial number ready.

For any problems reported to the IBM Support Center, a Problem Management Record (PMR) is created. A PMR is an online software record used to keep track of software problems reported by customers.

• The IBM Support Center representative will create the PMR and give you its number.
• Have the SDR information you collected earlier handy because it may be needed for inclusion in the PMR.
• Record the PMR number. YOU WILL NEED IT to send data to the IBM Support Center. YOU WILL ALSO NEED IT on subsequent phone calls to the IBM Support Center to discuss this problem.
• Write the PMR number on ALL media you created in the previous steps, even if you are not going to send this data to the IBM Support Center at this time. The Support Center may request the data at a later time, so you want to ensure that neither the media nor the PMR number corresponding to it is lost.
• To send the media to the IBM Support Center, use this address:

  IBM Mailing Address
  IBM RS/6000 Scalable POWERparallel Systems Dept. 39KA, M/S P961, Bldg. 415 2455 South Road Poughkeepsie, N.Y. 12601-5400 ATTN: APAR Processing

  Note:

  If you are using multiple packages or envelopes to send the media, be sure to label them in a series, such as "1 of 5", "2 of 5", and so forth.

Be sure that the person you identified as your contact can be reached at the phone number you provided in the PMR.