05/22/96 Recovery from LED 552, 554, or 556 in AIX 4.1 SPECIAL NOTICES Information in this document is correct to the best of our knowledge at the time of this writing. Please send feedback by fax to "AIXServ Information" at (512) 823-4009. Please use this information with care. IBM will not be responsible for damages of any kind resulting from its use. The use of this information is the sole responsibility of the customer and depends on the customer's ability to eval- uate and integrate this information into the customer's operational environment. WARNING: Please do not use this document if the system is a /usr client, diskless client, or dataless client. ABOUT THIS DOCUMENT This document discusses the known causes of LED 552, 554 and 556. Included is a procedure for recovery from these errors. This document is applicable to AIX 4.1. CAUSES OF AN LED 552, 554, OR 556 The known causes of an LED 552, 554, or 556 during IPL on a RISC System/6000 are: o A corrupted file system. o A corrupted journaled-file-system (JFS) log device. o A bad IPL-device record or bad IPL-device magic number. (The magic number indicates the device type.) o A corrupted copy of the ODM (Object Data Manager) data- base on the boot logical volume. o A fixed disk (hard disk) in the state of inactive in the root volume group. RECOVERY PROCEDURE To diagnose and fix the problem, you will need to boot from BOOTABLE MEDIA and run fsck (file-system check) on each file system. If the file-system check fails, you may need to perform other steps. WARNING: Please do not use this document if the system is a /usr client, diskless client, or dataless client. 1. Turn the key to the Service position. 2. With BOOTABLE MEDIA OF THE SAME VERSION AND LEVEL AS THE SYSTEM, boot the system. The bootable media can be ANY ONE of the following: Recovery from LED 552, 554, or 556 in AIX 4.1 1 05/22/96 o Bootable cdrom o Mksysb o Bootable Install Tape Follow the prompts to the "Welcome to Base OS" menu. 3. Choose "Start Maintenance Mode for System Recovery" (Option 3) The Next Screen would prompt the following Menu "Mainte- nance" o Choose "Access a Root Volume Group" (Option 1) At this stage the console will display information about rootvg, and also Display a Menu with two Options. o Choose "Access this volume group and start a shell before mounting the filesystems" (Option 2) If you get errors from the above option, do not continue with the rest of this procedure. Correct the problem causing the error. If you need assistance correcting the problem causing the error, contact one of the fol- lowing: o local branch office o your point of sale o 1-800-CALL-AIX (to register for fee-based services) All of the above avenues for assistance may be billable. | 4. Run the following commands to check and repair file systems. (The "-y" option gives fsck permission to repair file systems when necessary.) fsck -y /dev/hd1 fsck -y /dev/hd2 fsck -y /dev/hd3 fsck -y /dev/hd4 fsck -y /dev/hd9var If fsck indicates that o block 8 could not be read, the file system is prob- | ably unrecoverable. See step 5 for information on unrecoverable file systems. o block 8 could be read, but one of the following errors is given: "fsck: Not an AIXV3 file system." "fsck: Not a recognized file system type." | then go to step 6. Recovery from LED 552, 554, or 556 in AIX 4.1 2 05/22/96 o a file system has an unknown log record type, or | fsck fails in the logredo process, then go to step | 7. | o IF THE FILE SYSTEM CHECKS WERE SUCCESSFUL, skip to | step 9. 5. The easiest way to fix an unrecoverable file system is to recreate it. This involves deleting it from the system and restoring it from a backup. Note that hd2 and hd3 can be recreated but hd4 cannot be recreated. If hd4 is unrecoverable, you must reinstall AIX. For assistance with unrecoverable file systems, contact your local branch office or point of sale, or call 1-800-CALL-AIX to register for fee-based support ser- vices. DO NOT FOLLOW THE REST OF THE STEPS IN THIS FAX. 6. Attempt to repair the file system with this command: dd count=1 bs=4k skip=31 seek=1 if=/dev/hd# of=/dev/hd# Replace "hd#" with the appropriate file system. | Now skip to step 8. 7. A corruption of the JFS log logical volume has been detected. Use the logform command to reformat it. /usr/sbin/logform /dev/hd8 Answer YES when asked if you want to destroy the log. | 8. Repeat step 4 for all file systems that did not success- | fully complete fsck the first time. If step 4 fails a | second time, the file system is almost always unrecover- | able. See step 5 for an explanation of the options at | this point. In most cases, step 4 will be successful. | If step 4 is successful, continue to step 9. 9. With the key in Normal position, run the following com- mands to reboot the system: exit sync;sync;sync shutdown -Fr As you reboot in Normal mode, NOTICE HOW MANY TIMES LED 551 APPEARS. If LED 551 appears twice, fsck is probably | failing because of a bad fshelper file. If this is the | case and you are running AFS, see step 12. The majority of LED 552s, 554s, and 556s will be resolved at this point. If you still have an LED 552, 554, or 556, try the following steps: | 10. Repeat steps 1 through 3. 11. Run the following commands, which will remove much of the system's configuration, saving it to a backup direc- tory. Recovery from LED 552, 554, or 556 in AIX 4.1 3 05/22/96 mount /dev/hd4 /mnt mount /dev/hd2 /usr mkdir /mnt/etc/objrepos/bak cp /mnt/etc/objrepos/Cu* /mnt/etc/objrepos/bak cp /etc/objrepos/Cu* /mnt/etc/objrepos /etc/umount all exit Determine which disk is the boot disk with the lslv command. The boot disk will be shown in the PV1 column of the lslv output. lslv -m hd5 Save the clean ODM database to the boot logical volume. ('#' is the number of the fixed disk, determined with the previous command.) savebase -d /dev/hdisk# | If you are running AFS, go to step 12, otherwise, go to | step 13. 12. If you are running the Andrew File System (AFS), use the following commands to find out whether you have more than one version of the v3fshelper file. cd /sbin/helpers ls -l v3fshelper* If you have only one version of the v3fshelper file (for | example, "v3fshelper"), proceed to step 13. If there is a version of v3fshelper marked as original (for example, "v3fshelper.orig"), run the following com- mands: cp v3fshelper v3fshelper.afs cp v3fshelper.orig v3fshelper 13. WARNING: Please do not proceed further if the system is a /usr client, diskless client, or dataless client. Recreate the boot image (hdisk# is the fixed disk deter- | mined in step 13 or 14): bosboot -a -d /dev/hdisk# | 14. If you copied files in step 12, copy the AFS file-system helper back to v3fshelper: cp v3fshelper.afs v3fshelper 15. Turn the key to Normal position and run shutdown -Fr If you followed all of the above steps and the system still stops at an LED 552, 554, or 556 during a reboot Recovery from LED 552, 554, or 556 in AIX 4.1 4 05/22/96 in Normal mode, you may want to pursue further system recovery assistance from one of the following: o local branch office o your point of sale o 1-800-CALL-AIX (to register for fee-based services) All of the above avenues for assistance may be billable. For reasons of time and the integrity of your AIX oper- ating system, the best alternative at this point may be to reinstall AIX. Recovery from LED 552, 554, or 556 in AIX 4.1 5 05/22/96 READER'S COMMENTS Please fax this form to (512) 823-4009, attention "AIXServ Informa- tion". You may also e-mail comments to: elizabet@austin.ibm.com. These comments should include the same customer information requested below. Use this form to tell us what you think about this document. If you have found errors in it, or if you want to express your opinion about it (such as organization, subject matter, appearance) or make sug- gestions for improvement, this is the form to use. If you need technical assistance, contact your local branch office, point of sale, or 1-800-CALL-AIX (for information about support offer- ings). These services may be billable. Faxes on a variety of sub- jects may be ordered free of charge from 1-800-IBM-4FAX. Outside the U.S. call 415-855-4329 using a fax machine phone. When you send comments to IBM, you grant IBM a nonexclusive right to use or distribute your comments in any way it believes appropriate without incurring any obligation to you. NOTE: If you have a problem report or item number, supplying that number may help us determine why a procedure did or did not work in your specific situation. Problem Report or Item #: Branch Office or Customer #: Be sure to print your name and fax number below if you would like a reply: Name: Fax Number: ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ END OF DOCUMENT (led552.41.krn., 4FAX# 4188) Recovery from LED 552, 554, or 556 in AIX 4.1 6