08/29/95 7135 RAIDiant Array --Common Installation Issues SPECIAL NOTICES Information in this document is correct to the best of our knowledge at the time of this writing. Please send feedback by fax to "AIXServ Information" at (512) 823-4009. Please use this information with care. IBM will not be responsible for damages of any kind resulting from its use. The use of this information is the sole responsibility of the customer and depends on the customer's ability to eval- uate and integrate this information into the customer's operational environment. +----------------------------------------------------------+ | | | NOTE: The information in this document has NOT been | | verified for AIX 4.1. | | | +----------------------------------------------------------+ ABOUT THIS DOCUMENT This document contains setup tips for the IBM RAIDiant Array 7135 model 110. This document applies to AIX 3.2.5. INSTALL THE RAIDIANT ARRAY SOFTWARE FIRST You should install the RAIDiant Array software before attaching the 7135 to your RISC System/6000. If you have already booted with the 7135 attached before you installed the RAIDiant Array software, extra SCSI disk drives will have been configured into the system. Remove them by doing the following: 1. List all configured 7135 hdisks with "lsdev -Ct array". 2. Remove disks from the configuration with "rmdev -l -d". 3. Power off the 7135 after removing the drive definitions from the system and leave it off until you have installed the RAIDiant Array software. After installing the software you can attach the 7135, power it on and boot the RISC System/6000. RAIDIANT ARRAY SOFTWARE VERSIONS AND UPDATES Once the software is installed, you can determine which version you have with "lslpp -l scarray.obj". If you get a line of output showing "scarray.obj 03.02.00.00" (the 3.2 level) you have the code that supports active-passive controllers. If you have "scarray.obj 7135 RAIDiant Array --Common Installation Issues 1 08/29/95 03.03.00.00" or "scarray.obj 03.03.00.01" (the 3.3 level) you have the code that supports dual-active controllers. 3.3.0.1 level is required to support the 2.2 GB or 4.5 GB disk drives. We recommend that you get the dual-active code as it provides better performance if you have two control- lers and has design changes to prevent problems that can occur in the active-passive code. If you do not have installation media for scarray.obj, or you have active- passive and would like dual-active code, you can order it free by calling IBM Direct (800-IBM-CALL) and asking for feature code (FC) 5059 for AIX. To get to 3.3.0.1 requires that you install the dual-active code and several fixes listed below. For all 7135 installations, we recommend these fixes (note that the latest superceding fix is in parenthesis): U433447 (U435254) SMIT support for 7135 U434467 (U437507) Diagnostic support for 7135 U435115 Device Driver/Diagnostics enhancements U436790 (U437507) Device Driver/Diagnostics enhancements U436971 mkvg fix for PVs with > 1016 PPs/PV For 7135s connected via the SCSI-2 Differential F/W adapter the following fixes are required: U436450 (U437739) SCSI-2 Fast/Wide Device Driver U429285 SCSI-2 Fast/Wide Differential Enhancement For the scarray.obj 3.2 active-passive code: U432241 (U436353) 7135 fixes For 2.2 GB and 4.5 GB disk drive support (scarray.obj 3.3 only): U437007 7135 Maintenance Package 3.3.0.1 U437287 7135 and iostat fixes You can check to see if you have these fixes with "lslpp -lB ...". Either a state of APPLIED or COMMITTED means that it is installed, and a PTF number in parenthesis indicates a superceding PTF. If you ordered AIX with FC 5059 after 5/26/95, you will have these fixes on your install tape. Be sure you have a copy of the correct version of the doc- umentation. For scarray.obj 3.2, you should have the "READI and REACT Software Guide for the IBM 7135 RAIDiant Array Model 110" SC23-2572-01. For scarray.obj 3.3, you should have "7135 RAIDiant Array for AIX, Installation and Refer- ence" SC23-2728-01. 7135 RAIDiant Array --Common Installation Issues 2 08/29/95 VERIFY THE RISC SYSTEM/6000 SEES THE 7135 CORRECTLY If your RISC System/6000 configures the 7135 correctly, you will have a disk array router (dar) device for each 7135. For each disk array controllers (dac), you will have a dac device. And for each LUN, you will have an hdisk device. Each bank of 5 physical drives in the 7135 comes preconfig- ured as a RAID-5 LUN. For example, if you have a 7135 with two controllers and 15 disk drives, and you have one internal disk drive in your RS/6000 (hdisk0), you will have a dar0, dac0, dac1, hdisk1, hdisk2 and hdisk3 as follows: dac0 Available 00-03-00-00 7135 Disk Array Controller dac1 Available 00-04-00-10 7135 Disk Array Controller dar0 Available 7135 Disk Array Router hdisk1 Available 00-03-00-00 7135 Disk Array Device hdisk2 Available 00-03-00-01 7135 Disk Array Device hdisk3 Available 00-03-00-02 7135 Disk Array Device Use the following commands to list these devices: lsdev -C | grep Array (lists all the 7135 devices) lsdev -Ct dar (lists only the dar) lsdev -Ct dac7135 (lists only the dacs) lsdev -Ct array (lists only the 7135 hdisks) All of these devices should be in the "Available" state (this assumes that the devices are powered on and connected to the RISC System/6000). If you have devices in the "Defined" state, you may have inaccurate information in the customized device database and need to correct it. For example, if the 7135 was attached to the RISC System/6000 before the RAIDiant Array software was installed, you will have additional "Defined" hdisks. Phantom or ghost disks can also be created when the 7135 is originally attached to an adapter in one slot and is later attached to another slot. Use the following procedure to clean up the database (if you all the devices do not show up in the Available state, you may have a hardware problem): 1. Remove all the 7135 information from the device database using one of the following methods: o With the following script: lsdev -Ct array | cut -d ' ' -f1 | xargs -n1 rmdev -d -l lsdev -Ct dar | cut -d ' ' -f1 | xargs -n1 rmdev -d -l lsdev -Ct dac7135 | cut -d ' ' -f1 | xargs -n1 rmdev -d -l o Manually: a. Remove all 7135 and non-existent hdisks using "rmdev -l -d" for each hdisk where is one of the disks you want to remove from the database. b. Remove all dars using "rmdev -l -d" for each dar. 7135 RAIDiant Array --Common Installation Issues 3 08/29/95 c. Remove all dacs using "rmdev -l -d" for each dac. o Using smit (with the 3.3 code only) follow the menu: # smit array_menu Disk Array Controller Configuration Remove a Disk Array Subsystem dar0 Available 7135 Disk Array Router KEEP definition in database [no] 2. Reconfigure the devices using "cfgmgr" or reboot the system. The system will "walk the bus" and configure the devices it finds. 3. Verify that the devices configured correctly using "lsdev -C | grep Array". 4. Update the boot logical volume using "savebase -v" so it contains the correct device information. Note that for scarray.obj 3.2, running "cfgmgr" or rebooting does not always put the 7135 hdisks in the Available state (e.g., after the hdisks have become Defined due to a hardware problem or after a reinstall), and you must run "ace -z dac0" to get them into the Available state. VERIFY SCSI CABLING The SCSI-2 Fast/Wide Differential Adapter has two SCSI busses: internal and external. You cannot have drives that are part of rootvg attached to the same adapter as a 7135. If the system does not see all the array controllers, or they come up in the Defined state, you may have a cabling problem. Use the following procedure to make sure the SCSI adapter can communicate to the array controller (this trans- mits a SCSI inquiry command to a selected SCSI address and listens for a response). 1. Determine the SCSI adapter location address and the array controller SCSI IDs. If the array controller is configured, you can use the "lsdev -Cc array" command. For the example above: dac0 Available 00-03-00-00 7135 Disk Array Controller dac1 Available 00-04-00-10 7135 Disk Array Controller dac0 is connected to the SCSI adapter with location code 00-03-00, and dac0's SCSI ID is 0 (the 7th digit). 2. If you can't list the dacs, you must physically check it's SCSI ID (you must lower the fans on the back of the 7135 controller and then you can see the windows on the left showing the SCSI IDs) and determine which adapter it is cabled to. The default and preferred SCSI ID switch settings are 0 and 1. 7135 RAIDiant Array --Common Installation Issues 4 08/29/95 3. Varyoff all 7135 volume groups using "varyoffvg ". 4. Put the 7135 hdisks and dars in the Defined state with "rmdev -l ". 5. Run "diag" and choose the following: Service Aids SCSI Bus (press Enter twice) 6. Select the SCSI adapter attached to one of the dacs. 7. Press F4 and choose the SCSI ID of the dac. 8. Press F7 to commit and send the SCSI inquiry. If the command fails, there is a problem with either the SCSI cabling, the SCSI adapter, or the dac. RAID LEVELS The 7135 comes preconfigured with each bank of five disks configured as a RAID 5 LUN; thus, from AIX's point of view it looks like a 7.67 GB hdisk (assuming you have 2 GB drives in the 7135). We expect most customers will use the default configuration. Due to the RAID 5 write penalty, you should expect slower performance than you would have with standard SCSI disk drives. The write penalty is a result of the fault tolerance RAID 5 provides. We don't expect many customers to use RAID 3. It is not possible to create a volume group with RAID 3 hdisks on a 7135; consequently, you cannot create a file system and regular AIX files and your application will have to use raw I/O to the hdisk using RAID 3. RAID 0 stripes the data across the drives in the LUN, thus providing higher performance, but no fault tolerance. RAID 1 mirrors the data across two drives. Note that the AIX LVM provides mirroring capability without using a 7135. Mirroring with the 7135 offloads the CPU and AIX from doing the mirroring. VOLUME GROUPS AND LOGICAL VOLUME GUIDELINES When you create a volume group (VG), generally you need to use a physical partition size greater than the default. The LVM limits the number of physical partitions (PPs) to 1016 in a physical volume (PV). The number of default 4 MB PPs required for the 8 GB default hdisk in a 7135 would exceed the 1016 limit. No warning will be given for this condition unless you have U436971 installed in which case it will not allow you to create a VG violating this limit. You cannot change the PP size in a volume group without deleting the volume group and recreating it. You can check how many PPs are in your 7135 hdisks by running "lspv 7135 RAIDiant Array --Common Installation Issues 5 08/29/95 " and looking at the "TOTAL PPs:" field. This field should not exceed 1016. In general, you should not mix hdisks with different RAID characteristics in the same volume group. If you mix RAID 0 and RAID 5 hdisks, and a file system has partitions on both hdisks, and you lose a drive in the RAID 0 LUN, you will have lost all your data in the file system. Be aware that changing many parameters of the LUN will cause you to lose all the data on the LUN. AIX does not know how to handle a hdisk that changes size. Consequentially, you cannot easily change a LUN once it is loaded with data. For example, if you want to increase a LUN's size, you will have to backup all the data, unmount and remove the file systems, remove the hdisk from the volume group, change the LUN, add the hdisk back to the volume group, recreate and mount the file systems and restore the data. If performance is important, you should plan your volume groups and logical volume placement before you create them because recreating them and restoring your data can be time consuming. Intra-disk and inter-disk allocation policies still apply for RAID hdisks. However, if you implement sub- luns, creating a filesystem in the "center" of the disk is actually creating a filesystem in the center of the sub-lun which may be on the edge of the physical disks. You can get performance planning assistance from the AIX Support Fami- ly's Consult Line service. MICRCODE Microcode is code that is run by hardware other than a CPU, and for the 7135, we are concerned with microcode for the SCSI adapters, disk array controllers (dac), physical disks in the 7135, and for the EEPROMs in the 7135. This micro- code is typically loaded as a file on the RISC System/6000 and is downloaded to the device. For the SCSI 2 differen- tial adapters, the code is downloaded when the system boots. For the Fast & Wide SCSI adapter, the dacs and the 7135 disks, you download the code by using diagnostic service aids from AIX. Incompatible levels of microcode are a common source of problems. Ensure that the latest microcode you have avail- able on your system is installed for all the 7135 compo- nents. After changing any microcode, or the EEPROM, you must shut- down the RISC System/6000, power off the 7135, and reboot for changes to take effect. You can change the microcode for all the devices and reboot once. Generally, the order of assuring and updating the microcode is as follows: 1. SCSI Adapters (scsi or ascsi) 2. Disk Array Controller (dac) 3. EEPROM 7135 RAIDiant Array --Common Installation Issues 6 08/29/95 4. Physical Disks WARNING: Do not run diagnostics on the 7135 until all micro- code has been loaded and the 7135 has been powered off and on. The microcode files are described below. Keep in mind that the file names may change (generally to higher numbered file names) as improvements are made. SCSI ADAPTER MICROCODE The 7135 is attached to a SCSI-2 Differential (narrow) adapter or to a SCSI-2 Differential Fast & Wide adapter. Each dac can be connected to a separate adapter on separate SCSI buses, or both dacs can be connected to the same adapter on the same SCSI bus. You can list the SCSI adapters on your system with "lsdev -Cc adapter | grep scsi". NARROW ADAPTER For the narrow adapter (type 4-2) determine the level of microcode with "lsattr -El -a ucode" where is the scsi adapter attached to the 7135, e.g., "lsattr -El scsi1 -a ucode". You will see the microcode file listed. The latest supported file is /etc/microcode/8d77.a0.16. FAST & WIDE ADAPTER For the Fast & Wide adapter(s) (type 4-6) use "lscfg -vl ascsi*" and check the ROS Level and ID field for a value of 72, which is the microcode level. We recommend that you do not attach internal disks to this adapter if it is also con- nected to a 7135. If you do not have a microcode level of 72, and you have the 7135 connected to more than one RS/6000, you will need ECA 192 (ordered by the CE), or for the SP2 ECA 008, that provides the microcode on a diskette. If your 7135 is attached to only one RS/6000, a microcode level of 70 or 72 is OK. Use the documentation that comes with the ECA to install the microcode. The microcode has a file name of /etc/microcode/8EFC3011.72M. You can use the following procedure if (i) you have the dosread command installed on the system, (ii) you have PTF U429285 or U429287 installed, AND (iii) the adapter is not used as a boot adapter: 1. Varyoff any volume groups using the 7135. 2. Put the child devices of the ascsi in the Defined state using "rmdev -l " including any 7135 hdisk, dar, dac, vscsi and tmscsi devices. 3. Run "diag". 4. Choose "Service Aids". 5. Choose "Microcode Download". 7135 RAIDiant Array --Common Installation Issues 7 08/29/95 6. Choose the adapter to which you want to download the microcode. 7. Choose "Functional Microcode". 8. Choose "Download the latest level of microcode". 9. Select "Yes" to the question "do you have the microcode diskette?" 10. Insert the diskette into the diskette drive and press Enter. 11. After the message indicating success, you should also run advanced diagnostics, system verification on the adapter. 12. Shutdown and reboot the system for the new microcode to take effect. DISK ARRAY CONTROLLER MICROCODE To determine the level of microcode on the dac, run lscfg -vl dac0 | grep Z2 You will see a line similar to Device Specific.(Z2)........010010 For scarray.obj 3.2, the numbers should be 010010, and, for scarray.obj 3.3, the numbers should be 020604 or 020605. The 020604 microcode file is stored on the RISC System/6000 as /etc/array/fw/7135.02.01. The 020605 file is /etc/array/fw/7135.02.06. We recommend the latest level of microcode. If you have installed U437007, have 2.2 or 4.5 GB drives in your 7135, you must have the 020605 level of microcode installed on the dac. If your microcode is not correct, you can download the latest level you have on disk with the following procedure: 1. Varyoff any volume groups using the 7135. 2. Run "diag". 3. Choose "Service Aids". 4. Choose "7135 RAIDiant Array Service Aids". 5. Choose the active array controller (usually dac0). 6. Choose "Array Controller Microcode Download". 7. Choose "Download microcode from the hard disk". 8. Choose "Download the latest level of microcode". 7135 RAIDiant Array --Common Installation Issues 8 08/29/95 If you get an error, try removing the hdisks, dar, and dacs with "rmdev -l -d". Then run cfgmgr and try again. 7135 EEPROM SCARRAY.OBJ 3.2 To update the EEPROM with scarray.obj 3.2: 1. First, varyoff any 7135 volume groups. 2. Run the following command: /usr/lpp/scarray/seteeprom /dev/ where is one of the 7135 hdisks. 3. You should download the microcode to the EEPROMs on both dacs. If you have two array controllers, you need to switch the active controller and run the command again. You can switch the active controller with "smit react", or, e.g., run the following command to switch from dac0 to dac1. (This command does not apply for scarray.obj 3.3.) /etc/methods/chgdar -l dar0 -a pri_controller=dac1 SCARRAY.OBJ 3.3 To set the EEPROM with scarray.obj 3.3: 1. Varyoff any 7135 volume groups. 2. Run "diag" and choose Service Aids 7135 RAIDiant Array Service Aids Update EEPROM 3. Then choose the active dac and choose "YES" to download it. This will download the /etc/array/fw/7135eeprom.01 file to the EEPROM. 4. Be sure to update both array controllers with this pro- cedure if you have two of them. If you are installing scarray.obj 3.3 and have not installed scarray.obj 3.2, to update the EEPROM file on the second controller you will have to run the following command. chdev -l dar0 This will make both controllers active (i.e., dual-active). At this point, you can update the EEPROM on the second con- troller. 7135 RAIDiant Array --Common Installation Issues 9 08/29/95 7135 PHYSICAL DISK MICRCODE For scarray.obj 3.2, to download the disk microcode, run "diag", and choose "Service Aids", "7135 RAIDiant Array Service Aids", "dac0" or "dac1" whichever is the active con- troller, and "Spare Physical Disk Microcode Download". In scarray.obj 3.3, the procedure is the same except on the last menu the choice is titled "Physical Disk Microcode Download". You should download the disk array controller microcode and update the EEPROM before downloading the phys- ical disk microcode. There is no method to determine what microcode is downloaded on the disks. You have to download the microcode to each disk in the 7135 separately. It is OK to download this code more than once. The microcode for 3.2 is in the file: /etc/array/fw/0664M1H.A0900617.00020520 For scarray.obj 3.3, two files exist: 0664M1H.A0900617.00020540 0664M1S.A0900617.00020540 These are compatible with the 020604 level of dac microcode. The new microcode will work with scarray.obj 3.2, but 3.3 requires the new microcode. The 020605 level of dac micro- code is compatible with physical disk microcode with filenames ending in 5A0 rather than the 540 listed previ- ously. For the 2.2 GB and 4.5 GB drives, no physical disk microcode files exist on the system; consequentially, no microcode download is required for these drives. BACKUPS Even though the 7135 provides fault tolerance, you should still backup your data. Note that a "mksysb" backup will not backup data that is not in the root volume group. Sysback/6000's full system backup will backup data in all volume groups. However, since the 7135 device drivers do not run in maintenance mode, you can only restore data from the 7135s after the rootvg has been restored and the system has booted. If you have to reinstall a system with a 7135, Sysback or mksysb, power off the 7135 during the boot from tape, otherwise you may have extra "Other SCSI Disk" drives defined in the system database, and your hdisk names may be out of order after the reinstall. 7135 RAIDiant Array --Common Installation Issues 10 08/29/95 READER'S COMMENTS Please fax this form to (512) 823-4009, attention "AIXServ Informa- tion". You may also e-mail comments to: elizabet@austin.ibm.com. These comments should include the same customer information requested below. Use this form to tell us what you think about this document. If you have found errors in it, or if you want to express your opinion about it (such as organization, subject matter, appearance) or make sug- gestions for improvement, this is the form to use. If you need technical assistance, contact your local branch office, point of sale, or 1-800-CALL-AIX (for information about support offer- ings). These services may be billable. Faxes on a variety of sub- jects may be ordered free of charge from 1-800-IBM-4FAX. Outside the U.S. call 415-855-4329 using a fax machine phone. When you send comments to IBM, you grant IBM a nonexclusive right to use or distribute your comments in any way it believes appropriate without incurring any obligation to you. NOTE: If you have a problem report or item number, supplying that number may help us determine why a procedure did or did not work in your specific situation. Problem Report or Item #: Branch Office or Customer #: Be sure to print your name and fax number below if you would like a reply: Name: Fax Number: ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ END OF DOCUMENT (7135.inst.tips.325.ha_, 4FAX# 3140) 7135 RAIDiant Array --Common Installation Issues 11