Diagnosis Guide

Error symptoms, responses, and recoveries

If your system shows signs of a switch failure, locate the symptom and perform the recovery action described. All the recovery actions described require that the user have root access on the specified switch node. If any of the recovery actions fail to resolve your problem, contact IBM Support Center.

Note:

If your system is running in restricted root access mode, the following commands must be issued from the control workstation:

• CSS_test
• Efence
• Eprimary
• Equiesce
• Estart
• Eunfence
• Eunpartition
• mult_senders_test
• switch_stress
• wrap_test

SP Switch2 symptoms and recovery actions

Table 44 lists the known symptoms of a failure in the SP Switch2, and points the user to the location of the detailed diagnostics and recovery action. You may have a symptom that does not appear in the table. In this case, view the error entry in the AIX error log and see AIX Error Log information.

Table 44. SP Switch2 symptoms and recovery actions

Recover an SP Switch2 node

You can restart the fault_service_Worm_RTG_CS daemon on the node by issuing:

/usr/lpp/ssp/css/rc.switch 

Following the rc.switch, run this command to determine if the daemon is still running or has died:

ps -e | grep Worm

At this point you should be able to Eunfence the node by issuing:

Eunfence problem_node_number

The output should be similar to the following:

All nodes successfully unfenced.

If you cannot resolve the problem, contact the IBM Support Center. You should also attempt to gather all log files associated with this failure. See Information to collect before contacting the IBM Support Center.

Worm error recovery

The following steps enable you to recover from switch initialization failures that impact the worm subsystem.

1. Login to the switch primary node. See the Eprimary command to determine which node is the switch primary node.
2. Run the command errpt -a |pg and search for message Estart failed label CS_SW_ESTRT_FAIL_RE. The Detail Data will give the return code of the failure:

   switch initialization failed with xx
   

   where xx is the number to look up in Table 45.

3. If you did not find the error log entry, view the bottom of the file /var/adm/SPlogs/css0/p0/flt or /var/adm/SPlogs/css1/p0/flt. Look for a message similar to:
   • CSworm_bfs_phase1() failed with rc=xx
   • CSworm_bfs_phase2() failed with rc=xx

   where xx is the number to look up in Table 45.

4. Use the rc (return code) to retrieve the appropriate entry in Table 45.
5. If the return code cannot be found in the table, or the actions taken did not correct the problem, contact the IBM Support Center.

Table 45. SP Switch2 worm return codes and analysis

Ecommands error recovery

Error isolation for any of the Ecommands is as follows.

1. View the error output returned from the command. Note the error message number and text.
2. Find the message in PSSP: Messages Reference.
3. More information can be obtained from the Ecommands.log trace file, see Ecommands.log.
4. Perform the recommended recovery action.
5. If the Ecommand failed because it was unable to communicate with every node, see Diagnosing SP Security Services problems.
6. If the Ecommand failed because it cannot access the SDR, or the SDR is set up incorrectly, see Verify the System Data Repository (SDR).
7. After the recovery action is taken, if the problem persists, see Information to collect before contacting the IBM Support Center and contact the IBM Support Center.

Estart error recovery

To isolate and recover from failures in the Estart command, follow these steps:

1. Login to the primary node.
2. View the bottom of the file /var/adm/SPlogs/css0/p0/flt or /var/adm/SPlogs/css1/p0/flt.
3. Use the failure message as an index to Table 46.
4. If the failure message cannot be found in the table, or the actions taken did not correct the problem, contact the IBM Support Center.

Table 46. SP Switch2 Estart problems and analysis

Unfence an SP Switch2 node

The recovery action to take depends on the current status of the switch, and the personality of the switch node to be unfenced. The SP Switch2 status is limited to whether it is operational or not. The personality of the switch node to unfence is whether or not the node is to become the primary node, primary backup node, or a secondary node of the switch. For more information on any of the commands used in this section, see PSSP: Command and Technical Reference.

To display the switch primary node and primary backup node on all switch planes, issue the command:

Eprimary

Example output is:

plane 0:  none  - primary
plane 0:  5     - oncoming primary
plane 0:  none  - primary backup
plane 0:  45    - oncoming primary backup
plane 0:  1     - autounfence
 
plane 1:  none  - primary
plane 1:  5     - oncoming primary
plane 1:  none  - primary backup
plane 1:  45    - oncoming primary backup
plane 1:  1     - autounfence

In this example, no primary node is available. Therefore, the SP Switch2 is not operational. Remember that the Efence and Eunfence commands work only when the Eprimary command shows a valid node number as the primary node.

Note:

In the following commands, the -p flag can be used to specify the number of the switch plane. If the -p flag is not used, the command is applied to all switch planes.

1. SP Switch2 is operational and the node is to be the secondary. The node to unfence is not listed as the primary or the oncoming primary, and there is a primary node.

   Use the command Eunfence to unfence the node.

2. SP Switch2 is operational and the node is to be the primary. The node to unfence is the oncoming primary, and another node is currently the primary.
   • Use the command Eunfence to unfence the node.
   • Use the command Estart to set the node to its primary personality.
3. SP Switch2 is not operational and the node is to be the secondary. No node is listed as primary, and another node is listed as the oncoming primary.
   • Use the command Estart to operate the SP Switch2.
   • Use the command Eunfence to unfence the node.
4. SP Switch2 is not operational and the node is to be the primary. No node is listed as primary and the fenced node is listed as the oncoming primary.
   • Use the command Eprimary to set another node as oncoming primary.

     Select a node that is not fenced as an oncoming primary, otherwise the Estart command with fail again.

   • Use the command Estart to operate the SP Switch2.
   • Use the command Eunfence to unfence the node.
   • Use the command Eprimary to set the unfenced node to be the oncoming primary.
   • Use the command Estart to set the node as the switch primary.

Eunfence error recovery

This section is used to help you when you failed to unfence your node, following the unfence procedure described in Unfence an SP Switch2 node.

To isolate and correct most Eunfence problems, you should refer first to Ecommands error recovery.

Note:

In the following commands, the -p flag can be used to specify the number of the switch plane. If the -p flag is not used, the command is applied to all switch planes.

The following list provides additional reasons for a particular node to fail to Eunfence:

1. The Eunfence of a node failed, but the SP Switch2 was not Estarted. You cannot attempt to Eunfence any node on an SP Switch2 that is not started. Issue the Estart command.
2. The node can no longer be reached through the switch network. More information can be gathered from the out.top trace file, see out.top.
3. The SP Switch2 node failed to Eunfence because the switch topology could not be distributed. See Diagnosing SP Security Services problems.
4. The node failed to respond when attempting to Eunfence it. See SP Switch2 node diagnostics to isolate and correct the problem.
5. The user receives the message "Cannot Unfence node xxx - timeout", the most likely cause is that the fault service daemon (fault_service_Worm_RTG_CS) is not running on the node. If the this is the case, issue the /usr/lpp/ssp/css/rc.switch command to start the daemon. If the daemon is still not running, refer to the rc.switch.log trace file. See rc.switch.log.
6. The user receives a message similar to "Cannot Unfence node xxx - timeout", and you have replaced the switch cable. See Cable diagnostics. Even though the fault service daemon (fault_service_Worm_RTG_CS) is running, you must issue the /usr/lpp/ssp/css/rc.switch command to reload and reset the adapter before you can try to Eunfence the node.
7. If any of the preceding procedures fail to resolve the problem, and the node is still fenced, gather the css logs of the primary node and the fenced node. This can be accomplished by logging into those nodes and issuing the /usr/lpp/ssp/css/css.snap command. See Information to collect before contacting the IBM Support Center.

switch_responds is still on after node panic

This section addresses the case where a node panics, host_responds becomes 0 and switch_responds0 or switch_responds0 are still 1. This is a valid condition when the SP Switch2 adapter, on the crashed node, has no outstanding requests to or from this node. The SP Switch2 is now in a state where it can become backlogged, since the link is still marked as up. This can cause problems on other parts of the SP Switch2 network.

Each node fault-service daemon is responsible for updating its switch_responds0 and switch_responds1. The SP Switch2 primary node detects fallen links and turns off the appropriate switch_responds The switch_responds0 or switch_responds1 is turned on only during Estart or Eunfence command processing. A node panic with switch_responds 1 is a legitimate occurrence. There are two cases:

1. With primary or backup SP Switch2 nodes running, the switch_responds0 or switch_responds1 is updated only after a packet is sent to the panicked node. Therefore, a user can change switch_responds0 or switch_responds1 by trying to ping the panicked node. Having HA (hats and hags) run on the nodes can remedy the situation, since they run IP packets between the nodes casually in order to check the links (LAN Adapter event).
2. Without primary or backup SP Switch2 node running, there is no switch control. In this case, switch_responds0 or switch_responds1 is not updated. Only a new Estart command, specifying the -p flag with the correct switch plane number, can correct this.