IBM* PCI Hot-Plug Controller Driver v1.07G for SuSE 8.0/SuSE SLES 8.0/ RedHat 2.1 Advanced Server/RedHat 7.3 and Installation Instructions. Readme.txt Version 1.07G CONTENTS --------- 1.0 Hot-Plug PCI Installation Overview for SuSE 8.0/SuSE SLES 8.0 and RedHat 2.1 Advanced Server/Redhat 7.3 1.1 Overview 1.2 Limitations 1.3 Dependencies: 2.0 Change History 3.0 Installation and Setup Instructions for SuSE 8.0/SuSE SLES 8.0 and RedHat 2.1 Advanced Server/Redhat 7.3 3.1 Kernel configuration 3.1.1 For SuSE 8.0/SuSE SLES 8.0 3.1.2 For RedHat 2.1 Advanced Server 3.1.3 For RedHat 7.3 3.2 Driver installation 4.0 How to perform Hotplug PCI Operations (command-line) 4.1 Required steps in Hotplug Operations 4.1.1 Removing an adapter 4.1.2 Adding a new adapter to an empty slot 4.1.3 Changing the state of attention LED 4.1.4 Other files in the slot directory 4.2 Warnings about incorrect usage of Hotplug PCI. 5.0 Installation and Setup instructions for Graphical User Interface 6.0 Troubleshooting hot plug PCI operations 6.1 Basic Troubleshooting 6.2 Attention Indicator LED 6.3 Interrupt handling 1.0 Hot-Plug PCI Installation Overview for SuSE 8.0/SuSE SLES 8.0 and RedHat Advanced Server 2.1/Redhat 7.3 1.1 Overview The Hot Plug PCI support for SuSE 8.0/SuSE SLES 8.0 and RedHat 2.1 Advanced Server/RedHat 7.3 provided by IBM is compliant with many different specifications, including full compliance with the PCI Hot-Plug v2.2 specification. The IBM Hot Plug PCI system support consists of an interlock switch and a set of two LED's for each hot plug PCI slot. One LED will remain on when power is ON to the given slot. The other LED will indicate that attention is required to that slot or will be used to indicate which slot was chosen for a hotplug action. Slots that do not have these devices are NOT hot plug slots. All hotplug operations must be controlled either through a command line interface provided by the Linux kernel or through an appropriate GUI (see Section 5.0 'Installation and Setup instructions for Graphical User Interface). IN NO CASE SHOULD PHYSICAL HOT PLUG OPERATIONS BE DONE WITHOUT FIRST REMOVING POWER FROM THE SLOT. IF A SLOT DOES NOT HAVE A LATCH, DO NOT REMOVE THE ADAPTER. DOING SO CAN CAUSE SERIOUS DAMAGE TO THE SYSTEM AND ADAPTER. The order of events in any hot-plug operation are: - Load the kernel PCI hotplug core driver. - Mount the PCI hotplug file system. - Load the IBM PCI hotplug controller driver to provide hot plug PCI support. - For hot-add operation, open the latch and insert an adapter and any necessary cabling. Close latch. - At command prompt, echo the requested values into the hotplug file system to power on or power off the desired PCI slots. - For hot-remove operation, When the adapter power is OFF, open the latch and remove an adapter and any necessary cabling. Close latch. NOTE: The power to a slot cannot be turned ON until the latch is closed for the given slot. The position of the latch (opened or closed) is not important if no adapter is in the given slot. - If selected, the adapter power is turned ON and the adapter is configured. At this point, you may need to manually load the drivers associated with the newly added adapter card. 1.2 Limitations - NOTE: the limitation below was removed after driver version 0.6, or RPM version 1.2. However, if no Remote Supervisor Adapter is present in the system and/or no module for Service Processor is present (ibmasm), limitation still exists. Therefore, to be able to load/unload ibmphp driver multiple times, make sure ibmasm module is loaded in the system. Cannot load the IBM PCI hotplug controller driver multiple times within same boot session. If the driver is loaded, adapters are hot-added or hot-removed, then the driver is unloaded and re-loaded, erroneous behavior will occur. - You need to be logged on as a root user in order to be able to operate on the slots either via command-line or GUI. - On x-Series 440, if the latch to a slot was open during the boot of the server, then hot-adding adapter cards to that slot subsequently returns the following error (after about 1 minute of time out): "setting bus speed failed. was not able to set the bus." To fix the problem, open the latch, wait 2 seconds, close the latch, and send the command again. - This driver Will not support onboard PCI-PCI bridges. Additionally, only adapters cards with a single bus behind its onboard bridge are supported. - This driver does not support hot-adding/removing a nested bridge device, a bridge by itself, or any graphical devices. - This driver will not function properly if there is no APIC in the system. - Opening the latch while the adapter card is active may cause the system to reboot or hang. Under no circumstances should the latch be opened while power is applied to an active PCI slot. - Multi-function PCI devices are supported as long as the multi-function capability is not provided with or through a PCI-to-PCI bridge. - Devices that are not PCI v2.2 compliant or that do not implement the PCI presence detection pins are not supported. - Non-PCI devices are not hot pluggable. 1.3 Dependencies: - All files that are required by the Hotplug controller driver are provided in the standard distribution system and included RPM (ibmphp-'VERSION_NUMBER'-[OS Version].rpm). NOTE: If you want to be able to load/unload the IBM* PCI Hot-Plug Controller Driver, make sure you install ibmasm RPM and that ibmasm module is loaded at least when you are trying to load/unload ibmphp driver. 2.0 Change History of ibmphp ------------------------------ ************************* * v0.6-1.2A Changes * * November, 2002 * ************************* - Added support for one Hot-Plug controller to be able to support more than 4 buses ************************* * v0.6-1.2A Changes * * October, 2002 * ************************* Cosmetic changes in this version: - 0.6-1.2A indicates this is from RPM version 1.2A, since the driver in the kernel will look different than in the RPM - For SuSE 8.0, we have added updated pci_hotplug module, and as a result, had to change the slot names. For example, chassis_1_slot_2_bus_9, or rxe_1_slot_1_bus_20. Moreover, some files (such as bus name and max_adapter_speed) had to be dropped (because of changed pci_hotplug module), while others, such as cur_bus_speed and max_bus_speed had to change the format they present the information. Man pages also had to be changed to display up-to-date information. ************************* * v0.6 Changes * * July, 2002 * ************************* - Added PCI support for Hot-Plug controller - Changed pci_hotplug core to report more information to the user. (Unfortunately, changes can only be applied to SuSE since RedHat configurations statically include pci_hotplug module) - Fixed polling - Added man pages - Added functionality to store information into NVRAM (ability to load/unload ibmphp in same boot session). This functionality is dependent on ibmasm module present in the system. (You may rpm ibmasm*.rpm in order to get the module) - Changed the slot names to be [chassis#/rxe#]slot# - Added blue light functionality into a separate GUI ************************* * ibmphp v0.5 * * June, 2002 * ************************* - Added support for x-Series 235 - Fixed some bugs - Release of binary RPM for RedHat 7.3 ************************* * v0.2 Changes * * March/April, 2002 * ************************* - Added bus speed limitations - Fixed polling - Added support for RXE100 - Release of RPMs for Suse 8.0 ************************* * v0.1 Changes * * January 14, 2002 * ************************* - Made formatting changes to comply with the Linux community. - Updated the readme file to remove incorrect information as well as to add additional instructions and clarifications. ************************* * v1.00 Changes * * December 14, 2001 * ************************* - Original Release 3.0 Installation and Setup Instructions ----------------------------------------- NOTE: To successfully install provided RPM, make sure that the source code for linux kernel tree is installed on the system. You can get the source code by finding the correct RPM kernel-source-'VERSION-NUMBER'.rpm. Also make sure the source code matches your kernel version. 3.1 Kernel configuration ------------------------- 3.1.1 For SuSE 8.0/SuSE SLES 8.0 -------------------------------- If SuSE 8.0/SuSE SLES 8.0 is installed on IBM x-series 440/445/235/360/255, the kernel options 64GB-SMP are selected, therefore, before installing ibmphp RPM, you will need to do the following: rpm -qa | grep kernel Sample result (NOTE: Results will vary with different kernel versions): kernel-source-2.4.18.SuSE-35 SuSE is configured with the Yast tool. Use it install the appropriate kernel source files. After installation configure the kernel sources by executing the following commands: cd /lib/modules/`uname -r`/build make clean cp -f /boot/vmlinuz.version.h include/linux/version.h cp -f /boot/vmlinuz.config .config make cloneconfig make dep 3.1.2 For RedHat 2.1 Advanced Server ------------------------------------ rpm -qa | grep kernel Sample result (NOTE: Results will vary with different kernel versions): kernel-headers-2.4.2-2 kernel-source-2.4.2-2 kernel-smp-2.4.2-2 kernel-2.4.2-2 Note: On x-Series 440/445, make sure you are running summit kernel. Make sure that kernel-headers and kernel-source rpm packages are present and that they match your kernel version number. In other words, make sure kernel-headers-VERSION_NUMBER and kernel-source-VERSION-NUMBER have the same VERSION-NUMBER as your currently running kernel. (If you have summit kernel installed, and you type uname -r at the prompt, the result might look something like 2.4.9-e.3summit. The VERSION-NUMBER in this case is 2.4.9-e.3summit). You will also need install the source code for the summit kernel: 1. Install the kernel-.src.rpm from RedHat Advanced Server 2.1 CD #4 Ex, rpm -ivh kernel-2.4.9-e.3.src.rpm 2. cd /lib/modules/`uname -r`/build 3. patch -p1 < /usr/src/redhat/SOURCES/linux-2.4.9-summit.patch NOTE: if you have done step #3 previously once before, you need not do it again. In such a case, an error will be displayed with the following: 'Reversed patch detected...' and will ask you to either answer 'y' or 'n'. You can type c to exit out of the interactive menu. 4. make mrproper 5. cp configs/kernel-2.4.9-i686-summit.config .config 6. Go to the directory where ibmphp RPM is stored. Type the following to extract a file: rpm2cpio ibmphp-src-redhat-'VERSION_NUMBER'-2.1AS.rpm | cpio -ivd ./usr/src/redhat/SOURCES/Makefile.script Make sure your bash version is > 2.0. You can verify that by typing bash --version at the prompt. Type the following to launch the script: ./usr/src/redhat/SOURCES/Makefile.script Note: This script modifies the Makefile in /lib/modules/`uname -r`/build directory to match your running kernel. The command above (rpm2cpio) created a file tree in the directory where your RPM is contained. To remove it, type command: rm -rf ./usr MAKE SURE YOU ARE IN THE SAME DIRECTORY YOU WERE WHILE YOU ISSUED RPM2CPIO COMMAND, AND NOTICE THE LEADING . (dot) IN FRONT OF /usr IMPORTANT: Make sure you are NOT in the / directory. (you can verify which directory you are in by typing pwd at the command prompt). 7. cd /lib/modules/`uname -r`/build 8. make menuconfig Choose and save configuration file. Execute the following command: make dep 3.1.3 For RedHat 7.3 ------------------------------------- Install RedHat 7.3 and latest kernel errata using RPM. Make sure you install kernel-source and rpm-build packages, and that they match your kernel version number. Type the following commands at the prompt: rpm -qa | grep kernel rpm -qa | grep rpm Sample result (NOTE: Results will vary with different kernel versions): kernel-smp-2.4.18-3 kernel-source-2.4.18-3 Once you have the appropriate kernel sources installed, you will need to configure them. Execute the following commands: cd /lib/modules/`uname -r`/build make mrproper ls configs Examples of config files you might see: configs/kernel-2.4.18-i686.config configs/kernel-2.4.18-i686-smp.config NOTE: Combination of `uname -r` and `uname -m` will help you decide which configuration file you should copy. Then copy the configuration file: cp configs/ .config Go to the directory where ibmphp RPM is stored. Type the following to extract a file: rpm2cpio ibmphp-src-redhat-'VERSION_NUMBER'-7.3.rpm | cpio -ivd ./usr/src/redhat/SOURCES/Makefile.script Make sure your bash version is > 2.0. You can verify that by typing bash --version at the prompt. Type the following to launch the script: ./usr/src/redhat/SOURCES/Makefile.script Note: This script modifies the Makefile in /lib/modules/`uname -r`/build directory to match your running kernel. The command above (rpm2cpio) created a file tree in the directory where your RPM is contained. To remove it, type command: rm -rf ./usr MAKE SURE YOU ARE IN THE SAME DIRECTORY YOU WERE WHILE YOU ISSUED RPM2CPIO COMMAND, AND NOTICE THE LEADING . (dot) IN FRONT OF /usr, IMPORTANT: Make sure you are NOT in the / directory. cd /lib/modules/`uname -r`/build make oldconfig make dep 3.2 Driver installation ------------------------ Note: If this is an upgrade to an existing IBM PCI Hotplug driver, then previously installed packages MUST first be removed and unloaded. A. Unload: Type the following two commands at a shell prompt to remove the old driver source and binaries (if you have them): To find out which ones your system currently has, type rpm -qa | grep ibmphp at the command prompt. If ibmphp module is currently loaded (you can see this by typing lsmod at the command prompt. You can type /sbin/ibmphpdown (for versions >= 1.2), or reboot the system (for earlier versions). You can also perform this operation after installing the new version of the driver (via rpm -ivh ...) B. To remove, type: NOTE: you may not have all of those packages installed on your system. rpm -e ibmphp 1. Go to the directory where ibmphp rpm package is stored, at the command prompt, type rpm -ivh ibmphp-'VERSION_NUMBER'-[8.0/sles8.0/2.1AS/7.3].rpm This rpm will compile the needed files, put the binary object files in the /lib/modules/... directory and add a comment into /etc/fstab for mounting (if needed). The hotplug directory is located by default at /hotplug. It will also load the driver. This RPM will install the man pages, so if anytime you want to find out information about the driver, you can type: man ibmphp at the command prompt. The RPM will also configure the needed files so that ibmphp will be loaded at bootup. 2. Add the ability to use the separate debug option. This can be done by adding the following line to /etc/syslog.conf file using your favourite editor underneath a line that starts with *.info or simply at the end of file: "*.debug /var/log/debug" In order for this change to take effect, please type the following commands at the prompt: touch /var/log/debug In RedHat: service syslog restart In SuSE: rcsyslog restart NOTE: The above section is only required to be performed prior to the initial use of the driver. This section can be skipped after the first installation/compilation. NOTE: If you want to use GUI instead of command line interface, you will need to rpm ibmphp-gui-'VERSION_NUMBER'-[8.0/sles8.0/2.1AS/7.3].rpm instead (if available), please see step "5.0 Installation and Setup instructions for Graphical User Interface" for details. To install the IBM PCI hotplug driver, perform the following: /sbin/ibmphpup NOTE: To view any errors due to improper operation, please view /var/log/messages or type "dmesg" at the command prompt. NOTE: If there is an unexpected failure, please power down the machine, reboot, and reload the IBM PCI hotplug driver with an additional debug option. Place a diskette into the floppy drive to save the error logs. The debug information can then be copied and saved for debugging purposes: /sbin/ibmphpup --debug In RedHat: mount /mnt/floppy cp /var/log/messages /mnt/floppy/messages.txt dmesg >& /mnt/floppy/dmesg.txt cp /var/log/debug /mnt/floppy/debug.txt umount /mnt/floppy In SuSE: mount /media/floppy cp /var/log/messages /media/floppy/messages.txt dmesg >& /media/floppy/dmesg.txt cp /var/log/debug /media/floppy/debug.txt umount /media/floppy To unload ibmphp driver, type /sbin/ibmphpdown NOTE: To be able to unload the ibmphp driver (and load it once again in the same boot session), you MUST have ibmasm module loaded, and Remote Supervisor Adapter present. Make sure your system has Remote Supervisor Adapter present. To find out if you have the necessary packages installed on your system, type rpm -qa | grep ibmasm at the command prompt. If you do not see it listed, please go to the IBM support website and get the latest RPM for Remote Supervisor Adapter (ibmasm*.rpm) and install this RPM on your system. If ibmasm is listed as part of the above instruction, make sure it is loaded. You can type lsmod to find out, and you can load it by typing insmod ibmasm at the prompt. 4.0 How to perform Hot-Plug PCI Operations (command line interface) NOTE: You can always type man ibmphp at the command prompt if you want to read this section from your system. 4.1 Required steps in Hot-Plug Operations Make sure you are logged on as root to be able to issue hot-plug operations. NOTE: Opening an adapter latch will turn off the power to the slot. If a device driver was loaded for that slot, the system will most likely hang, requiring the system to be rebooted. 4.1.1 Removing an adapter 1. Verify that the pci_hotplug and ibmphp drivers are loaded. This can be done by issuing the "lsmod" command. (Note: On RedHat systems you will not see pci_hotplug listed as a result of lsmod command. However, if ibmphp loaded with no errors, and /hotplug filesystem is mounted (see #2) is a good indication pci_hotplug driver is loaded). 2. Verify that the /hotplug file system is mounted. This can be done by issuing the "mount" command. 3. Issue the power off command to the desired PCI hotplug capable slot. "echo 0 > /hotplug//power" Ex. "echo 0 > /hotplug/2/power" to turn power off to slot 2. NOTE: with RPM verion 1.2A and later, slot number is now chassis_#_slot_#_bus_# if the slot is located in the server, and rxe_#_slot_#_bus_# if the slot is physically located in the PCI expansion box (RXE100). If you are using Graphical User Interface, you will see bus # only in the 'Detailed slot information' tab. bus# is the bus id number on which the slot resides. 4. Verify that the power LED for the slot is OFF. If the LED is still ON and the attention LED is OFF, go back to the command line and verify the proper command was issued. NOTE: NEVER REMOVE AN ADAPTER FROM A SLOT WITH THE SLOT POWER STILL ON. THIS COULD RESULT IN A SYSTEM HANG AND/OR SERIOUS DAMAGE TO THE ADAPTER CARD AND/OR SYSTEM UNIT. 5. Open the latch and remove the adapter from the slot. 6. Close latch after adapter has been removed. 4.1.2 Adding a new adapter to an empty slot 1. Verify that the pci_hotplug and ibmphp drivers are loaded. This can be done by issuing the "lsmod" command. (Note: in RedHat, pci_hotplug module is statically linked, which means lsmod will not show it. However, if /hotplug file system is mounted, and ibmphp driver loaded with no errors, this is an indicator that pci_hotlug is loaded) 2. Verify that the /hotplug file system is mounted. This can be done by issuing the "mount" command. 3. Open the latch, place the new adapter in the desired slot and close the latch. 4. Attach any necessary cables that the newly inserted adapter may require. 5. Issue the power on command to the desired PCI hotplug capable slot where the new adapter will be inserted. "echo 1 > /hotplug//power" Ex. "echo 1 > /hotplug/chassis_1_slot_2_bus10/power" will turn power on to slot 2. 6. Verify that the power LED for the selected slot is now on. NOTE: If any error occurs, the IBM PCI hotplug controller driver will turn the attention LED on for the selected slot. To check the corresponding error message, please read /var/log/messages or /var/log/debug (if the --debug or -d ibmphp option was provided). 4.1.3 Changing the state of attention LEDs Note: In addition to internal indications that the operation is currently being performed or an error occured, ibmphp driver lets user change the state of attention LEDs. 1. To turn on attention LED for a slot, type at the command prompt echo 1 > /hotplug//attention 2. To turn off attention LED for a slot, at the command prompt type echo 0 > /hotplug//attention 3. To make the attention LED blink for a slot, at the command prompt type echo 2 > /hotplug//attention 4.1.4 Other files in the slot directory You can find the status of a slot by going going to the directory corresponding to that slot cd /hotplug// Each directory contains several files. You can do cat for each particular file. - latch file contains 1 if the latch of the slot is open, 0 if the latch is closed - attention file contains 0 if attention is off, 1 if attention is on, 2 if attention is blinking - power file contains 0 if the power is off, 1 if power is on - adapter file contains 1 if there is an adapter in the slot, 0 otherwise - in SuSE filesystem, there are other files present as well: - cur_bus_speed displays what the current bus speed is. 33 MHz PCI, 66 MHz PCI, 66 MHz PCI-X, 100 MHz PCIX, 133MHz PCIX, Unknown Bus Speed is displayed when the bus speed is 66 MHz, and the bus mode is unknown - max_bus_speed displays the maximum speed this bus is capable of running at. 33 MHz PCI, 66 MHz PCI, 66 MHz PCI-X, 100 MHz PCIX, 133 MHz PCIX. 4.2 Warnings about incorrect usage of Hotplug PCI. Below are several warnings about hot plug PCI operations: 1. Do not remove an adapter from a non-hotplug PCI slot. 2. Grounding equipment, including wrist straps, should be used when working inside any system unit to protect against electrostatic discharge (ESD) that could damage system components and adapter cards. 3. Make sure that any adapter card hot added to the system is fully seated before attempting to turn on power. 4. Opening an adapter latch will turn off the power to the slot. If a device driver was loaded for that slot, the system could hang, requiring the system to be rebooted. 5. Do NOT remove the plastic cover over the system components. Removal of the plastic cover exposes system components that may be shorted by adapter brackets during hot plug PCI operations. 5.0 Installation and Setup instructions for Graphical User Interface -------------------------------------------------------------------- Make sure you are logged on as root to be able to issue hot-plug operations. NOTE: In the future, a gui rpm will be provided that will install a customized version of Graphical User Interface tailored just for IBM servers. For now, you would need to download the latest version from http://www.kroah.com/linux/hotplug/ (for example, www.kroah.com/linux/hotplug/pcihpview-0.3-1.i386.rpm) 1. Make sure you have gtk and gtk-devel packages installed on your system 2. If this is an upgrade to an existing GUI interface, then previously installed packages MUST first be removed and unloaded. Type the following two commands at a shell prompt to remove the old packages (if you have them): To find out which ones your system currently has, type rpm -qa | grep pcihpview at the command prompt. To remove, type: rpm -e pcihpview 3. To configure and use the pcihpview, please type the following in the directory where pcihpview-0.4b-1.i386.rpm resides rpm -ivh pcihpview-0.4b-1.i386.rpm 4. This will produce an executable pcihpview. You need to be in X-environment in order to use it. To use it, type pcihpview /hotplug & The order of operations is the same as for command-line interface. To operate on a slot, highlight it and right-mouse click to see the available operations. Limitations: the current version of pcihpview (0.4) does not automatically update if a change has been detected from polling (such as a latch opening or closing, or if you issue a command through command line interface). However, you can choose Refresh option to refresh your screen. There is also a limit of 1 pcihpview running at the same time on a system. 6.0 Troubleshooting Hotplug PCI operations ------------------------------------------- 6.1 Basic Troubleshooting There are several troubleshooting techniques that can be used to determine why a hot plug PCI operation failed. Two LED's are provided for each hotplug PCI capable slot. The attention LED blinks to indicate that an operation is currently being performed and is solid if attention is required. The second LED indicates power state. Error messages are stored in /var/log/messages and /var/log/debug (if the debug=1 option was provided). 6.2 Attention LED The attention LED is controlled by the IBM PCI hotplug controller driver. The attention LED is used by the device driver to let the user know that an operation is currently ongoing on that particular slot or if an error has occurred. When powering a slot on, you will see the attention indicator flash momentarily while the newly inserted adapter is configured. The user can manually turn the attention LED on by issuing the attention on command: "echo 1 > /hotplug//attention" Ex. "echo 1> /hotplug/2/attention" will turn the attention LED on for slot 2. Similarly, the user can turn the attention LED off by issuing the off command: "echo 0 > /hotplug//attention" Ex. "echo 0 > /hotplug/2/attention" will turn the attention LED off for slot 2. 6.3 Interrupt handling Interrupts are handled in the following manner: - Interrupts are preserved from the MPS table located in the extended BIOS data area (EBDA) of the system.