05/22/96 Recovery from LED 552, 554, or 556 in AIX 3.1 or 3.2 SPECIAL NOTICES Information in this document is correct to the best of our knowledge at the time of this writing. Please send feedback by fax to "AIXServ Information" at (512) 823-4009. Please use this information with care. IBM will not be responsible for damages of any kind resulting from its use. The use of this information is the sole responsibility of the customer and depends on the customer's ability to eval- uate and integrate this information into the customer's operational environment. +----------------------------------------------------------+ | | | NOTE: The information in this document has NOT been | | verified for AIX 4.1. | | | +----------------------------------------------------------+ WARNING: Please do not use this document if the system is a /usr client, diskless client, or dataless client. WARNING: Do not use this procedure on an AFS fileserver. Doing so may remove all AFS fileserver data because fsck would be run on the AFS volumes. CAUSES OF AN LED 552, 554, OR 556 The known causes of an LED 552, 554, or 556 during IPL on a RISC System/6000 are: o A corrupted file system. o A corrupted journaled-file-system (JFS) log device. o A bad IPL-device record or bad IPL-device magic number. (The magic number indicates the device type.) o A corrupted copy of the ODM (Object Data Manager) data- base on the boot logical volume. o A fixed disk (hard disk) in the state of inactive in the root volume group. SUMMARY OF THE RECOVERY PROCEDURE To diagnose and fix the problem, you will need to boot from BOSboot diskettes or tape and run fsck (file-system check) on each file system. If the file-system check fails, you may need to perform other steps. Recovery from LED 552, 554, or 556 in AIX 3.1 or 3.2 1 05/22/96 STEPS WARNING: Please do not use this document if the system is a /usr client, diskless client, or dataless client. 1. Turn the key to the Service position. 2. With BOSboot diskettes or tape OF THE SAME VERSION AND LEVEL AS THE OPERATING SYSTEM, boot the system. +----------------------------------------------------------+ | | | WARNING: If you boot a 3.2 system with 3.1 media, or | | boot a 3.1 system with 3.2 media, then you will not be | | able to use the standard scripts (getrootfs or | | /etc/continue) to bring your workstation into full main- | | tenance mode. | | | | Moreover, performing the scripts on a 3.1 system with | | 3.2 boot media may actually remove some files and | | prevent your system from booting successfully in normal | | mode until missing files (/etc/mount and /etc/umount) | | are replaced on the disk. | | | +----------------------------------------------------------+ NOTES: a. For information on BOSboot diskettes, refer to InfoExplorer or order fax #2462 from 1-800-IBM-4FAX (or 415-855-4FAX outside the U.S. from a fax machine phone). b. If booting from diskettes, when you see LED c07, insert the next diskette. This may be the optional display extensions diskette (required for AIX 3.2.5) or the display diskette. c. If you have AIX 3.2.5, used a fddi network install, and selected that option for the install device, you will need the communications extensions diskette. If this diskette is not available, you can build it from another system with fddi installed or call your branch office for assistance. NOTE: If you get a 552, 554, or 556 on this step, the diskette or tape is bad, and the machine is trying to boot off the fixed disk. Try it again with new BOSboot diskettes or tape. Follow the prompts to the installation/maintenance menu. 3. Choose the maintenance shell (option 5 for AIX 3.1, option 4 for AIX 3.2). 4. Determine the hdisk# to use with the getrootfs or /etc/continue command. If you have only one disk, then "hdisk0" is the proper hdisk# to use. If you have more than one disk, do the following: Recovery from LED 552, 554, or 556 in AIX 3.1 or 3.2 2 05/22/96 o FOR AIX 3.2.4 OR LATER: Run getrootfs The output indicates the disk that should be used with getrootfs in the next step. o FOR AIX 3.1 TO 3.2.3E: Run lqueryvg -Atp hdisk# | grep hd5 for each hdisk# (hdisk0, hdisk1, etc.) until you get output that looks like: 00005264feb3631c.2 hd5 1 The exact output you get will be different but will follow the form large_number.x hd5 1 You may find more than one disk has this output. These will all be disks which belong to the rootvg volume group. You may use any of the disks identi- fied to be in rootvg in the following step. 5. Now access the rootvg volume group by running /etc/continue (for AIX 3.1) or getrootfs (for AIX 3.2). ('#' is the number of the fixed disk, determined in step 4.) For AIX 3.1 only, run /etc/continue hdisk# sh For AIX 3.2 only, run getrootfs hdisk# sh If you get errors from /etc/continue or getrootfs, do not continue with the rest of this procedure. Correct the problem causing the error. If you need assistance correcting the problem causing the error, contact one of the following: o local branch office o your point of sale o 1-800-CALL-AIX (to register for fee-based services) All of the above avenues for assistance may be billable. | 6. Run the following commands to check and repair file systems. (The "-y" option gives fsck permission to repair file systems when necessary.) Recovery from LED 552, 554, or 556 in AIX 3.1 or 3.2 3 05/22/96 fsck -y /dev/hd1 fsck -y /dev/hd2 fsck -y /dev/hd3 fsck -y /dev/hd4 For AIX 3.2 only, also run fsck -y /dev/hd9var If fsck indicates that o block 8 could not be read, the file system is prob- | ably unrecoverable. See step 7. for information on unrecoverable file systems. o block 8 could be read, but one of the following errors is given: "fsck: Not an AIXV3 file system." "fsck: Not a recognized file system type." | then go to step 8. o a file system has an unknown log record type, or | fsck fails in the logredo process, then go to step | 9. | o THE FILE SYSTEM CHECKS WERE SUCCESSFUL, SKIP TO STEP | 11. 7. The easiest way to fix an unrecoverable file system is to recreate it. This involves deleting it from the system and restoring it from a backup. Note that hd2 and hd3 can be recreated but hd4 cannot be recreated. If hd4 is unrecoverable, you must reinstall AIX. For assistance with unrecoverable file systems, contact your local branch office or point of sale, or call 1-800-CALL-AIX to register for fee-based support ser- vices. DO NOT FOLLOW THE REST OF THE STEPS IN THIS FAX. 8. Attempt to repair the file system with this command: (replace "hd#" with the appropriate file system) dd count=1 bs=4k skip=31 seek=1 if=/dev/hd# of=/dev/hd# | Now skip to step 10. 9. A corruption of the JFS log logical volume has been detected. Use the logform command to reformat it. On AIX 3.1 only, run /etc/aix/logform /dev/hd8 On AIX 3.2 only, run /usr/sbin/logform /dev/hd8 Answer YES when asked if you want to destroy the log. Recovery from LED 552, 554, or 556 in AIX 3.1 or 3.2 4 05/22/96 | 10. Repeat step 6 for all file systems that did not success- | fully complete fsck the first time. If step 6 fails a | second time, the file system is almost always unrecover- | able -- see step 7 for an explanation of the options at | this point. In most cases, step 6 will be successful. | In that case, continue to step 11. 11. With the key in Normal position, run the following com- mands to reboot the system: exit sync;sync;sync shutdown -Fr As you reboot in Normal mode, notice how many times LED 551 (that's right, 551, not 552) appears. If LED 551 appears twice, fsck is probably failing because of a bad fshelper file. If this is the case and you are running | AFS, see step 15. The majority of LED 552s, 554s, and 556s will be resolved at this point. If you still have an LED 552, 554, or 556, try the following steps: | 12. Repeat steps 1 through 4. | 13. For AIX 3.2, skip to step 14. For AIX 3.1 only, run the following commands, which will remove much of the system's configuration, saving it to a backup directory. exit mkdir /etc/objrepos/bak cp /etc/objrepos/Cu* /etc/objrepos/bak cp /etc/objrepos/boot/Cu* /etc/objrepos Determine which disk is the boot disk with the lslv command. The boot disk will be shown in the PV1 column of the lslv output. lslv -m hd5 Save the clean ODM database to the boot logical volume. ('#' is the number of the fixed disk, determined with the previous command.) savebase -d /dev/hdisk# | If you are running AFS, go to step 15. Otherwise, go to | step 16. 14. For AIX 3.2 only, run the following commands, which will remove much of the system's configuration, saving it to a backup directory. Recovery from LED 552, 554, or 556 in AIX 3.1 or 3.2 5 05/22/96 mount /dev/hd4 /mnt mount /dev/hd2 /usr mkdir /mnt/etc/objrepos/bak cp /mnt/etc/objrepos/Cu* /mnt/etc/objrepos/bak cp /etc/objrepos/Cu* /mnt/etc/objrepos /etc/umount all exit Determine which disk is the boot disk with the lslv command. The boot disk will be shown in the PV1 column of the lslv output. lslv -m hd5 Save the clean ODM database to the boot logical volume. ('#' is the number of the fixed disk, determined with the previous command.) savebase -d /dev/hdisk# | If you are running AFS, go to step 15. Otherwise, go to | step 16. 15. If you are running the Andrew File System (AFS), use the following commands to find out whether you have more than one version of the v3fshelper file. For AIX 3.1: cd /etc/helpers For AIX 3.2: cd /sbin/helpers Then in both AIX 3.1 and 3.2, ls -l v3fshelper* If you have only one version of the v3fshelper file (for example, "v3fshelper"), proceed to step 17. If there is a version of v3fshelper marked as original (for example, "v3fshelper.orig"), run the following com- mands: cp v3fshelper v3fshelper.afs cp v3fshelper.orig v3fshelper 16. WARNING: Please do not proceed further if the system is a /usr client, diskless client, or dataless client. | Recreate the boot image (hdisk# is the fixed disk deter- | mined in step 13 or 14.): bosboot -a -d /dev/hdisk# | 17. If you copied files in step 15, copy the AFS file-system helper back to v3fshelper: Recovery from LED 552, 554, or 556 in AIX 3.1 or 3.2 6 05/22/96 cp v3fshelper.afs v3fshelper 18. Turn the key to Normal position and run shutdown -Fr If you followed all of the above steps and the system still stops at an LED 552, 554, or 556 during a reboot in Normal mode, you may want to pursue further system recovery assist- ance from one of the following: o local branch office o your point of sale o 1-800-CALL-AIX (to register for fee-based services) All of the above avenues for assistance may be billable. For reasons of time and the integrity of your AIX operating system, the best alternative at this point may be to reinstall AIX. Recovery from LED 552, 554, or 556 in AIX 3.1 or 3.2 7 05/22/96 READER'S COMMENTS Please fax this form to (512) 823-4009, attention "AIXServ Informa- tion". You may also e-mail comments to: elizabet@austin.ibm.com. These comments should include the same customer information requested below. Use this form to tell us what you think about this document. If you have found errors in it, or if you want to express your opinion about it (such as organization, subject matter, appearance) or make sug- gestions for improvement, this is the form to use. If you need technical assistance, contact your local branch office, point of sale, or 1-800-CALL-AIX (for information about support offer- ings). These services may be billable. Faxes on a variety of sub- jects may be ordered free of charge from 1-800-IBM-4FAX. Outside the U.S. call 415-855-4329 using a fax machine phone. When you send comments to IBM, you grant IBM a nonexclusive right to use or distribute your comments in any way it believes appropriate without incurring any obligation to you. NOTE: If you have a problem report or item number, supplying that number may help us determine why a procedure did or did not work in your specific situation. Problem Report or Item #: Branch Office or Customer #: Be sure to print your name and fax number below if you would like a reply: Name: Fax Number: ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ END OF DOCUMENT (led552.krn, 4FAX# 1457) Recovery from LED 552, 554, or 556 in AIX 3.1 or 3.2 8