Some SP/2 tidbits ... --------------------------------------------------------------------------- To check one file on a bunch of nodes, dsh ls -l /etc/ssh2/sshd2_config | dshbak -c ;ls -l /etc/ssh2/sshd2_config To copy the CWS's version, pcp -p /etc/ssh2/sshd2_config /etc/ssh2/sshd2_config Of course, you can restrict it to specific nodes with the -wips01i,ips03i,ips05i,ips06i,ips07i option --------------------------------------------------------------------------- See http://www.rs6000.ibm.com/support/sp for info on different fix levels of PSSP code. --------------------------------------------------------------------------- Try spmon -d -G to see a snapshot of the whole SP/2. Returns stuff like ----------------------------- Frame 3 (and 1 & 2) ------------------------- Frame Node Node Host/Switch Key Env Front Panel LCD/LED is Slot Number Type Power Responds Switch Fail LCD/LED Flashing -------------------------------------------------------------------------- 1 33 extrn on no no N/A N/A LCDs are blank N/A --------------------------------------------------------------------------- To get a node to network boot, use nodecond 3 1 (for frame 3, node 1, e.g.) You do this when you want to install AIX on a node, presuming you've already done the spchvgobj -i bos.obj.ssp.432 -p PSSP-3.1 -v aix432 3 1 1 and spbootins -r install 3 1 1 To get a node to power off or on, try hmcmds off 33 (for node 33) or hmcmds on 3:1 (for frame 3, node 1) --------------------------------------------------------------------------- To Completely Reinstall AIX on a node (what I did for as0207 on 3/10/00) 1) spchvgobj -i bos.obj.ssp.432 -r rootvg -v aix432 -p PSSP-3.1 2 7 1 2) vi /etc/exports To remove /spdata/sys1/install exportfs -u /spdata/sys1/install 3) spbootins -r install 2 7 1 4) Start the install with nodecond 2 7 which network boots the node. 5) Follow the notes in my ~jasper/public_html/patent.new-machine.build.process.html --------------------------------------------------------------------------- To Upgrade AIX & PSSP on a Node, e.g. What I did for as0206 on 8/28/99, and as0201 on 9/09/99, and as0204 on 10/06/99, and as0205 on 12/01/99, but I had troubles that I never investigated, so I tried again on 4/03/00, This time I fixed it. and as0203 on 4/04/00, and as0202 on 4/05/00, 0) A good thing to do before you get going is to verify the SDR has the correct en0 ethernet adapter's MAC address by comparing a netstat -in | grep en0 | grep -i link on the node, with a splstdata -b frame# node# 1 | grep as0 on the CWS. If you want, you can even compare this with a lsnim -l as0nnne0 | grep if1 on the CWS. This just eliminates questions later should you have troubles. Another good thing to do is to check the error log on the node, fixing and cleaning out any old entries. Yet another good thing to do is to clear out root's mail, both already-inc'd mail (using scan, show, rmm) and yet-to-be-inc'd mail (inc it). ksh hint to remove a lot of mail, rmm $(scan | grep expression | awk '{print $1}' | tr -d '+') rm /Mail/inbox/,* 1) This is the command one uses with PSSP 3.1 instead of the spbootins command to set the AIX & PSSP level. spchvgobj -i bos.obj.ssp.432 -r rootvg -v aix432 -p PSSP-3.1 frame# node# 1 See the results of the command if you want to, with splstdata -b frame# node# 1 You'll see that next_install_image = bos.obj.ssp.432 lppsource_name = aix432 and the pssp_ver = PSSP-3.1 2) On the CWS, insure /etc/exports does not have the normal /spdata/sys1/install exported. There's a copy of the normal, everyday /etc/exports saved at /etc/exports.save, which has /cd and /spdata/sys1/install exported. This is what the CWS normally has exported, but when you run setup_server, it tries to export the /spdata/sys1/install/aix432/lppsource directory which, since it's a subdirectory of /spdata/sys1/install, get an NFS error. Use exportfs to see what's exported now, and vi /etc/exports exportfs -u /spdata/sys1/install to unexport whatever's now exported. You can keep /cd if you want. 3) spbootins -r migrate frame# node# 1 This sets the "response" field of the splstdata -b frame# node# 1 command to "migrate" and runs the setup_server command, which takes a few minutes to complete. At this point, check to see what's NFS exported with an exportfs command. For the as0204 install on 10/6/99, I had a lot of trouble here. It appeared setup_server wasn't doing the right thing with /etc/exports. What should be there is /spdata/sys1/install/aix432/spot/spot_aix432/usr /spdata/sys1/install/pssp/bosinst_data_migrate /spdata/sys1/install/pssp/pssp_script /export/nim/scripts/as0204e0.script All with -ro,root=as0204e0.patent.ibm.com,access=as0204e0.patent.ibm.com at the end, as well as /spdata/sys1/install/aix432/lppsource -ro I was getting different things, e.g. the lppsource line not having the extra stuff on it or missing lines altogether, which causes LED 611, I opened up PMR 24566,49R to investigate, but a week later, everything was working ok. If this situation occurs again, reopen the PMR or open a new one. The quickest way to debug, is to use a series of allnimres & unallnimres commands. allnimres -l 2 4 1, puts the lines in /etc/exports. unallnimres -l 2 4 1, takes them out presuming you've already done the spchvgobj & spbootins commands. 4) 4/3/2000 Note: I don't see any /etc/rc.sp anywhere that has this local mod I did long ago. I wonder what happened to it? Anyway, this bullet no longer applies since I lost my fix. If you want details, see my PMR 59782,49R dated 11/23/98, in my ~jasper/aixnotes/aix.support.center.history file. Beware of your local mod to /etc/rc.sp where you call /local/bin/lsof to insure inetd is indeed listening to kshell. The lsof in /local/bin doesn't work on AIX 4.3.2. I modified /etc/rc.sp to check the oslevel, and run /local/bin/lsof.for.AIX.4.3 instead. Insure the target node has either an unmodified /etc/rc.sp, or this fixed one. 5) Prepare the node for the DCE conversion. mkdir /var/dce/tmp /var/dce/config /var/dce/dced/backup Replace the rccleancred line in /etc/inittab, with cleanupdce:2:wait:/usr/bin/clean_up.dce > /dev/console 2>&1 and comment it out. Replace the rcdce line in /etc/inittab with rcdce:2:wait:/etc/rc.dce all > /dev/console 2>&1 # Start DCE/DFS Daemons and comment it out as well. Comment out the /etc/rc.local line as well, especially for something like the DB/2 nodes, so nothing gets started up. While you're at it, check for and delete if they exist, the following lines in /etc/inittab - db, i4ls, imnss, imqss, httpdlite Remove unneeded parms on the dfsd startup command, at the end of the /etc/dce/rc.dce file. Unneccessary parms are - callback $RPC_SUPPORTED_NETADDRS - blocks 20000 - cachedir /dfscache - chunksize 16 Reboot the machine/node to get an IPL without DCE/DFS up. When it's back up, you may have troubles getting to the IBM Internal network 'cause /etc/rc.local is commented out. You may need to dsh -was0nnne0 route add -net 9 -netmask 255.0.0.0 192.168.56.252 to get that routing back. Clear out the DFS cache directory. for i in $(jot 10 0);do rm /dfscache/V1$i*;done for i in $(jot 10 0);do rm /dfscache/V$i*;done find /dfscache -type f -exec rm {} \; Un-comment-out, i.e. restore the rcdce line in /etc/inittab. 6) Start the install with nodecond frame# node# which network boots the node. This will upgrade AIX & DCE on the node, but not SSP. Don't have a R/W spmon window open during this step, it'll cause it to fail. If you want to monitor things, from the CWS, tail -f /var/adm/SPlogs/spmon/nc/nc.2.5 (or whatever the latest file is in the /var/adm/SPlogs/spmon/nc directory). You can also monitor things with a R/O window via a s1term frame# node# command. Another handy way to monitor what's going on is to have a spled & window open on the CWS. On the 4/3/2000, as0205 conversion, this step hung after these lines in the /var/adm/SPlogs/spmon/nc/nc.2.5 file, Nodecond Status: command is boot /pci@fef00000/ethernet@10 Nodecond Status: network boot successfull Nodecond Status: bootp sent over ethernet Nodecond Status: waiting for "Welcome to AIX" message Nodecond Status: network boot proceeding, nodecond is exiting Nodecond Status: finished I opened a R/O s1term window to watch it more closely, and saw the bootp packets getting sent just fine, but when that image was booted, as0205 hung on LED E105. The http://rshelp.austin.ibm.com web site pointed me to PMR 48501,7TD which was an 8-week, Sev 1 problem that they were guessing at all kinds of things, then finally found an SP node, not involved with the problem at all, that had full duplex turned on for its private SP/2, BNC, 10Mbs, en0 interface, which is wrong. I checked my nodes with an appropriate dsh 'lsattr -E -l ent0 -a full_duplex 2>/dev/null' command, looking for full_duplex yes Full duplex True instead of full_duplex no Full duplex True I found that as0211 had full duplex turned on. I fixed as0211 by a chdev -l ent0 -a full_duplex=no command and as0205 installed just fine. Unfortunately, this was after I also wasted time updating the SDR 'cause a splstdata -a command showed nodes a lot of enet_rate & duplex settings to be null. The PMR said to insure they were set to 10 & half respectively. Using smitty, I changed this with a spethernt command, discovering 2 options not documented in my book, -f 10 & -d half, which are the defaults. I also fixed all the en1 interfaces, which at the time (4/4/00), were all 100/full duplex except for nodes 3-5 & 7-12 on the first frame. Now the splstdata -a command shows the right thing. I also tried applying service to the spot, but that didn't fix things either. First, restore the route to the IBM Internal network. Since /etc/rc.local is commented out, you'll need to dsh -was0nnne0 route add -net 9 -netmask 255.0.0.0 192.168.56.252 to get that routing back. 7) Check to insure DCE/DFS got upgraded ok. I used to have a lot of troubles here, but now I think things go pretty smoothly. If DFS isn't up already after the reboot, you'll probably see on the console if you've been watching it, or in the /etc/dce/cfgdce.log file, DCE Migration cannot be performed unless no DCE daemons are running. 0x11315b66: You can attempt to stop the daemons by running the command stop.dce, or you can stop them manually. Simply rerun /etc/rc.dce all and the conversion will finish and DCE & DFS will come up. When DCE & DFS are both there ok, configure dceunixd with a config.dce dce_unixd command. Restore the cleanupdce line that was commented out from /etc/inittab. 8) To upgrade the PSSP code, set the node's bootp response to customize, copy the PSSP 3.1 pssp_script to the node and run it. First, from the node itself, route add -net 9 -netmask 255.0.0.0 192.168.56.252 to restore the routing back to IBM's internal network. Then mkitab -i rc 'fbcheck:2:wait:/usr/sbin/fbcheck 2>&1 | alog -tboot > /dev/console # run /etc/firstboot' installp -u ssp.jm ssp.css Then, spbootins -r customize frame# node# 1 which will run setup_server, which takes a few minutes. From the CWS, type pcp -was0xxxe0 /spdata/sys1/install/pssp/pssp_script /tmp/pssp_script Now before you do this last step, check /tftpboot/script.cust on the CWS. Comment out the updating of everything except /etc/hosts, that is, at the end of the script, insure /etc/resolv.conf, /etc/passwd, /etc/group, /etc/security/passwd, /etc/security/group, and /etc/security/user are all commented out. I had to use ADSM to restore these files on 206. Then from the node, type /tmp/pssp_script or if you prefer, from the CWS, you can type dsh -was0xxxe0 /tmp/pssp_script If you want to watch the progress of this step, open another window and tail -f /var/adm/SPlogs/sysman/NODE.config.log.21428 where NODE is eg as0202e0 and 21428 is the PID of the ksh process that got forked off by the /tmp/pssp_script program. 9) After the install, check to see what you've got. For example, run oslevel -l 4.3.2.0 to see which filesets still need updating to get to AIX 4.3.2. For the 201, 204 & 206 updates, I've had to run installp -u bos.msg.en_US.diag.rte but be aware that you can't have 2 installp processes running at one time, so if you're still doing the /tmp/pssp_script step above, this step will have to wait. For the 201 & 204 updates, I also saw 3 devices.ssa.* filesets were still at the 4.2.1 level. I'm not sure why those filesets didn't get updated, probably because we used to have SSA drives on this system, but they got moved from when we moved the DB/2 database to as0209e0/1? I dunno. From the node, you can mount cws:/spdata/sys1/install/aix432/lppsource /mnt smitty update_all pointing it to the /mnt directory. Interestingly enough, this reinstalled ssp.jm & ssp.css. But these 3 filesets, Fileset Actual Level Maintenance Level --------------------------------------------------------------- devices.ssa.disk.rte 4.2.1.10 4.3.2.0 devices.ssa.IBM_raid.rte 4.2.1.7 4.3.2.0 devices.ssa.tm.rte 4.2.1.5 4.3.2.0 were a problem to install. Had to go to smitty installp and select "Install and Update from LATEST Available Software" at the bottom of the screen, and explicitly select the base 4.3.2.0 filesets for pointing it to the /mnt directory. and select these 3 guys, viz devices.ssa.IBM_raid.rte 4.3.2.0 devices.ssa.disk.rte 4.3.2.0 and devices.ssa.tm.rte 4.3.2.0 When you're done, umount /mnt. 10) Put the IBM message lines back in /etc/motd so that AIXCOPS will run clean. 11) Run lockdown.sh to insure things are still ok. It's not uncommon for lockdown to find ports that were previously either not defined or not running, now enabled. 12) Check for known errors, ls -l /etc | grep ' 0 ' # 8 zero-length files are ok. ls -l /usr/bin | grep ' 0 ' # There should be no zero-length files. ls -l /usr/sbin | grep ' 0 ' # There should be no zero-length files. ls -l /etc/services /etc/inetd.conf # Insure these are over 4,000 bytes big. grep -v '^#' /etc/inetd.conf # Should only be sshdfail, ssalld, kshell, klogin, and maybe clutild. grep '^start' /etc/rc.tcpip # Should only be syslogd, portmap, inetd, xntpd, and sshd. 13) Insure any 100Mbs ethernet interface (ent1 on SP/2 nodes, else probably ent0) is configured correctly for 100 Mb, full duplex. For unknown reasons, the initial boot has the en1 interface working ok, but subsequent reboots try configuring the interface to be 10 Mbs & half duplex. To see what the setting is in the ODM, type odmget -q 'name=ent1 AND attribute=media_speed' CuAt | grep value It should be value = "100_Full_Duplex" not value = "10_Half_Duplex" If it's wrong, you can set/fix it with chdev -l ent1 -a media_speed=100_Full_Duplex -P The -P option says to just change the ODM database. Since en1 is working for this boot, you don't have to change it. It's just the next boot you need to fix it for, and to do that, all you need do is change the ODM. If you need to, you can route add -net 9 -netmask 255.0.0.0 192.168.56.252 (Note these commands are different, due to it being a different ethernet adapter, than the E105 problems I describe in step 5) above.) 14) If this system has SSA on it, upgrade the SSA code with these commands, mount cws:/spdata/sys1/install/aix432/lppsource/PROD/SSA_Upgrades /mnt smitty installp Insure you select "Install and Update from ALL Available Software" at the bottom of the screen, else you won't get the SSA microcode filesets (ssamcode.* and ssadiskmcode.*). Point it to /mnt and go ahead and install everything, i.e. type "all" in the "SOFTWARE to install" field. Again, when you're done, umount /mnt To see what levels of microcode you have now, before they get updated, For each D40 enclosure, each SSA adapter, and each pdisk, lscfg -vl enclosure0 lscfg -vl ssa0 or lscfg -vl pdisk0 For each, Look at ROS Level and ID. See the http://www.hursley.ibm.com/ssa/rs6k web page for the latest levels for each type of device. The adapter microcode will automatically get updated the next time cfgmgr runs and the adapter & drives aren't being used. I.E. since the system is now quiesced, you can either run cfgmgr now (easiest), or shutdown -Fr. The enclosure or pdisk microcode, require a manual step. To update the enclosure or pdisk microcode, ensure all drives aren't in use, including, for an enclosure, any drives being used by other systems. Then - For each enclosure, type /etc/microcode/ssa_sesdld -d enclosure0 -f coral014.hex Even though the enclosure microcode is really updated, a lscfg -vl enclosure0 command will still show the downlevel code. To fix this, mkdev -l enclosure0 - To update all the drives at once, ssadload -u This takes about 30 seconds per drive. See my notes in ~jasper/aixnotes/ssa for further information. 15) Restore the /etc/rc.local line that was commented out from /etc/inittab. 16) shutdown -Fr --------------------------------------------------------------------------- To open a R/O window to a node's console, type s1term 2 13 To open a R/W window to a node's console, type spmon -open 2 13 --------------------------------------------------------------------------- To do a "Manual Node Conditioning", which I had to do when I got in the 4 new silver nodes and again when the S70 came in, Open a R/W window via spmon -open node23 for example, then power off the node, then power it on again. hmcmds off 49 hmcmds on 49 At the "memory keyboard network scsi ..." prompt, hit 1 to get into the "System Management Services" screens. For the Silver nodes, the sequence was Select 3 = Utilities, then 4 = Remote Initial Program Load Setup, For the S80, it was simply Select 3 = Remote Initial Program Load Setup, Then select the I.P. adapter and set the I.P. parameters as appropriate. Do the ping test. If the CWS is already set up to install this node, to boot the node from the Integrated ethernet adapter, Get back out the the main "System Management Services" menu Select 2 = Multiboot then 3 = Install From (6 on the S80) then "3 Ethernet ( Integrated )" On the S80, it was Select 6 = Multiboot then 4 = Select Boot Devices then 3 = Configure 1st Boot Device then 4 = Ethernet ( loc=U0.1-P1-I2/E1 ) "X" all the way out 'till you get the RS/6000 screen saying STARTING SOFTWARE, PLEASE WAIT... After AIX gets rewritten, the node will do the normal node-startup thing of sending the CWS a bootp packet to ask what it should do. --------------------------------------------------------------------------- The command spmon -gui calls under the covers to bring up the LED display is spled. --------------------------------------------------------------------------- To eliminate these errors in the error log of an SP/2 node, IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION 89B52AA5 0421114900 P S console SOFTWARE PROGRAM ERROR E85C5C4C 0421114500 P S CFGLFT SOFTWARE PROGRAM ERROR Type rmdev -dl lft0 rmdev -dl gxme0 rmdev -dl rcm0 then installp -ug devices.graphics.com Then you can errclear -N console 0 errclear -N CFGLFT 0 --------------------------------------------------------------------------- To refresh the hats/hags/haem, etc daemons in an SP complex, from the CWS type syspar_ctrl -r (refresh) Or for individual nodes, from the node itself, type /usr/lpp/ssp/bin/syspar_ctrl -k (kill) then /usr/lpp/ssp/bin/syspar_ctrl -s (start) --------------------------------------------------------------------------- There are four different microcodes associated with an SP/2, all are distinct and are updateable by the customer in various ways. They are 1) System Firmware: For each node. You can see the level you're at by a lscfg -vp command. Look for the first occurance of the "ROM Level (alterable) ..." line under the "System Firmware:" section. For example, ... Name: rom Model: AM29F040-PLCC Node: rom@fff00000 Physical Location: P2 System Firmware: ROM Level (alterable).......L98112 Version.....................RS6K System Info Specific.(YL)...P2 ... On 7-20-98, I updated this firmware for each of the 6 nodes in the second frame, by getting the WIL98112 IMGBIN file from AIXTOOLS, pcp-ing it to each node as /var/wil98112.img, then dsh-ing this shutdown command, shutdown -Fu /var/wil98112.img On 9-16-2000, I got a more recent version, WIL00125.IMG. I tried to copy it to all nodes' /var filesystem, but had troubles with a bunch of them. I only updated 216, since we were replacing the I/O planar. To extract the image from the CE's diskette, on the CWS, mkdir /tmp/fwupdate cd /tmp/fwupdate restore The procedure to update the node is the same as above. Copy the file to the node's /var filesystem, e.g. with pcp or scp, then tell the node to shutdown -Fu /var/WIL00125.IMG This shuts down and reboots the node. Updating the system firmware had to be done before updating the Service Processor Firmware, which I was also updating. See the notes below on that. 2) Service Processor Firmware: You can see the level of this firmware by the same lscfg -vp command. Look for the "ROM Level (alterable)..." line in the "IBM,sp" section. For example, ... Name: IBM,sp Node: IBM,sp@ie8 Physical Location: P2-X1 Service Processor: Part Number................. 93H4228 EC Level.................... E76324A FRU Number.................. 93H4214 Manufacture ID..............3966-1944843 ROM Level.(non-alterable)...19980414 Serial Number...............00007472 ROM Level (alterable).......19980414 ... On 7-20-98, I attempted to update this firmware on the new SP/2 nodes, but the version on AIXTOOLS in the 7025F50S PACKAGE file was SP980413 IMGBIN, and each node had level SP980414, so nothing got changed when I tried updating it. The procedure to update it by the way, is to get that SP980413 IMGBIN file copied to each node to /var/spflash.img, then invoke diag, - "Task Selection(Diagnostics, Advanced Diagnostics, Service Aids, etc.)" - "Update System or Service Processor Flash" near the bottom. - You can then point it to the file, /var/spflash.img to get it to update. On 9-16-2000, I got a newer version, wc991027.img. Similar to the System Firmware procedure, I copied this file to the node's /var file system, then told the node to shutdown -Fu /var/wc991027.img This shuts down and reboots the node. Yes, this was a double reboot we had to do, the first to update the System Firmware (see notes above) and the second to update this Service Processor Firmware. One last note, this reboot seemed to take a very long time, but it finally came back ok. 3) Node Supervisor Firmware: 4) Frame Supervisor Firmware: This is the firmware the PSSP code knows about and is now complaining that I should update. An "spsvrmgr -G -q status all" command returns spsvrmgr: Frame Slot Supervisor Media Installed Required State Versions Version Action _____ ____ __________ ____________ ____________ ____________ 1 0 Active u_10.3c.0709 u_10.3c.060c Upgrade u_10.3c.070c _____ ____ __________ ____________ ____________ ____________ 2 0 Active u_10.3c.0709 u_10.3c.070c None u_10.3c.070c ____ __________ ____________ ____________ ____________ 1 Active u_10.3e.0703 u_10.3e.0704 None u_10.3e.0704 ____ __________ ____________ ____________ ____________ 2 Active u_10.3e.0703 u_10.3e.0704 None u_10.3e.0704 ____ __________ ____________ ____________ ____________ 3 Active u_10.3e.0703 u_10.3e.0704 None u_10.3e.0704 ____ __________ ____________ ____________ ____________ 4 Active u_10.3e.0703 u_10.3e.0704 None u_10.3e.0704 ____ __________ ____________ ____________ ____________ 5 Active u_10.3e.0703 u_10.3e.0704 None u_10.3e.0704 ____ __________ ____________ ____________ ____________ 6 Active u_10.3e.0703 u_10.3e.0704 None u_10.3e.0704 I'm guessing that slot 0 refers to the Frame Supervisor, and the other slots 1-6 refer to the Node Supervisor. One can use smit to update. The above command was done after I had updated the CWS & Frame 2 to PSSP 2.4, but hadn't updated the Frame 1 nodes yet. --------------------------------------------------------------------------- When I got a sysplanar0 error on as0209, the Support Center had me go out to testcase.boulder.ibm.com and pick up /hardware/fromibm/9076/chrpdg.tar. This tool is used to collect data on processor hardware errors to deliver to the Support Center for debugging. Untar-ing this got me -r--r--r-- 3795 /tmp/petools/README.chrpdg -r-xr-xr-x 10625 /tmp/petools/chrpdg -rwxr-xr-x 5146 /tmp/petools/nvdump.chrp The README says the best way to run this chrpdg tool is to cd /tmp/petools ./chrpdg Running it gives the following messages ************************************************************************ ******************* CHRP Data Gather - version 3.0 ******************* ************************************************************************ Please enter the PMH and Branch Office information. (Example: 45345.b519) ====> 77076.49r If you don't know, hit Enter key (Example: 4B267810 25C38008 or 888-102-300-0C4) ====> ************************************************************************ Gathering the following information...please wait: ./chrpdg[99]: SDRGetObjects: not found. ./chrpdg[101]: nodenumber=nodenumberx*1: 0403-009 The specified number is not valid for this command. ... errpt ... errpt -a ... lscfg -vp ... lsdev -C ... lsresource -al ... Processor Attributes ... Memory Attributes ... STO and AIX Dump Settings ... CHRP device tree ... NVRAM Data ... checkstop/scanoutlog files ... diagnostic screens ... lslpp -ha ... instfix -i Data Gathering completed. Starting file compression... Compression completed! ************************************************************************ Please send the following tar file to the 'testcase.software.ibm.com' server and notify the Support Center via the PMH. Remember to transfer this file in BINARY FORMAT. Filename to send is located in ====> /tmp/77076.49r.000516.171406.tar.Z ftp file to testcase directory ====> /hardware/toibm/9076 ************************************************************************ All pretty self-explanatory. I dropped it off at testcase as they asked and do you know what? I was told that if you don't run this tool to collect the hardware error info before 7 days, it drops off. There was nothing they could do for me, but tell me to wait 'till it happens again. ---------------------------------------------------------------------------