Broadcom Advanced Server Program (BASP) Driver for Linux Version 6.2.1 ======================================================== BASP Driver for Linux, special release. CONTENTS -------- 1.0 Overview 1.1 Limitations 2.0 Change History 3.0 Build and Installation Instructions 3.1 Install - TAR archive 3.2 Install - RPM package 4.0 Configuration Information 4.1 Configuration - Red Hat & United Linux based Distribution 4.2 Configuration - Suse Distribution 4.3 Configuration and Startup for Other Linux Distribution 4.4 Startup scripts - Red Hat distributions 4.5 Configuration scripts - Red Hat and Suse 4.6 Broadcom NICE patches 4.7 Uninstall - RPM package 4.8 Removal of physical interface in Generic Trunking and 802.3ad mode 5.0 Web Sites and Support Phone Number 6.0 Trademarks and Notices 7.0 License and Disclaimer 1.0 Overview _____________ This package includes the Linux BASP driver support for all IBM Broadcom 570x-based Gigabit Ethernet,including 31P6301 and 22P7801 server adapter options. BASP is a kernel module designed for 2.4.x kernels that provides load-balancing, fault-tolerance, and VLAN features. These features are provided by creating teams that consist of multiple NIC interfaces. A team can consist of 1 to 8 NIC interfaces and each interface can be designated primary, or hot-standby (SLB team only). All primary NIC interfaces in a team will participate in Load-balancing operations by sending and receiving a portion of the total traffic. Hot-standby interfaces will take over in the event that all primary interfaces have lost their links. VLANs can be added to a team to allow multiple VLANs with different VLAN IDs. A virtual device is created for each VLAN added. BASP supports Smart Load-balance (SLB), SLB (Auto-Fallback Disable), Generic trunking and IEEE 802.3ad Link Aggregation. In both SLB modes and 802.3ad mode, all the NIC drivers must support Broadcom NIC Extension (NICE). In this release, several NIC drivers patched with NICE are included. * Both SLB modes work with all Ethernet switches without configuring the switch ports to any special trunking mode. Only IP traffic will be load-balanced in both inbound and outbound directions. * Generic trunking mode does not require NICE and can work with any NIC, however, it requires the Ethernet switch to support the technology and be properly configured. This mode is protocol-independent and all traffic should be load-balanced and fault-tolerant. * 802.3ad mode requires NICE drivers and Ethernet switches supporting IEEE 802.3ad Link Aggregation. This mode is protocol-independent and all traffic should be load-balanced and fault-tolerant. All the physical interfaces in the 802.3ad teams are defaulted to be LACP active. A 802.3ad team requires all the member NICs supporting NICE. All the member NICs, once in the 802.3ad team, will be set with the same MAC address. 1.1 Limitations ----------------- The current version of BASP has been tested on the latest Red Hat and United Linux distributions, as well as other similar Linux distributions for i386, ia64, and x86_64 CPU architectures using 2.4.x kernels. The module has been tested up to kernel version 2.4.20. Minor modification to the Makefile may be required if problems are experienced when compiling BASP on other i386 Linux distributions. VLANs are only supported by Broadcom NetXtreme Gigabit Ethernet. As opposed to VLANs support in other platforms, e.g. Windows and Netware, VLANs are not supported by Alteon Acenic driver (acenic.c). For full functionality, Broadcom driver for NetXtreme Gigabit Ethernet bcm5700 version 6.2.2 or newer should be used. DHCP address problem -------------------- We have identified that there is incompatiblity between SLB and IP address obtained via DHCP. In the lab, we have seen that the virtual network interface lost the IP address after running for a period of time. To solve the problem, always assign static IP address to the virtual network interfaces. HP2524 switch ------------- 802.3ad team member links disconnect and reconnect continuously when connects to the HP2524 switch. This is a 3rd party issue. It is seen only when configuring an 802.3ad team with greater than 2 members on the server and connecting an HP2524 switch, with lacp enabled as passive or active. The HP switch will show an lacp channel being brought up successfully with only 2 members. All other member's links will disconnect and reconnect. This does not occur with a Cisco Catalyst 6500. Spanning tree ------------- To avoid fail-over issues when using BASP, make sure that spanning tree is disabled on the switch that the network adapter is connected to. 2.0 Change History ____________________ The following problems have been fixed in the following versions: Version 6.2.1 ------------- - BASP SLB team sends stale arp requests after IP address of the team alias (e.g. sw0:0) has been changed. Version 6.2.0 ------------- - BASP panics occasionally during ifconfig up. Version 6.1.2 - SLB (Auto-Failback Disable) should be changed to SLB (Auto-Fallback Disable). Version 6.1.1 ------------- - SLB (Auto-Fallback Disable) team does not failback when all standbys fail. - Add default gateway option in setup scripts. Version 6.1.0 ----------------- - baspcfg show [] command not reading the parameter properly. - Add a new SLB mode that provides full manual failback Version 6.0.5 ------------- - BASP SLB team sends stale arp requests after IP address of the team has been changed. Version 6.0.4 ------------- - BASP teams will not start using RH7.3 Errata kernels for i386. Version 6.0.3 ------------- - BASP 6.0.2 freezes the system when trying to start SLB/GEC/AD teams w/VLANS. Seen on both Red Hat/ UL1.0 on all 3 system architectures (i386, ia64 x86_64). - When a BASP team is started, there are many warning messages, "protocol 0988 is buggy, dev eth0" logged into syslog. Version 6.0.2 ------------- - When configured with Intel Adapter in a GEC MVT team, disconnect the link cable causes network traffic drops. - Linux BASP igonres/drops multicast packets. - Remove version specific info from release notes. Generalize versions - BASP sample scripts no longer have write permission. - Update BASP to work with NAPI. Version 6.0.1 ------------- - Added RPM support for AMD 64-bit on SuSE 8.1 beta. - Added RPM support for Red Hat 8.1 beta. - Added RPM support for United Linux 1.0 based distributions. - Fixed potential duplicated MAC address problem in Generic Trunking or IEEE 802.3ad Link Aggregation mode. - Minor documentation cleanup. 3.0 Build and Installation Instructions _________________________________________ For installing RPM on Linux i386 distribution, follow instructions in "Install - RPM package" section. For installing TAR archive on i386, IA64 and AMD 64 distributions, follow instructions in "Install - TAR archive" section. 3.1 Install - TAR archive ------------------------- BASP for Linux is shipped in mixed forms, where the platform and kernel specific files are in source code, and the core file is in object form. Three packages are shipped in this release: two tar archives and two RPM package. "basplnx-{version}.i386.tgz" is the tar archive for i386 platform, "basplnx-{version}.ia64.tgz" is the tar archive for IA-64 platform, "basplnx-{version}.x86_64.tgz" is the tar archive for AMD 64 bit platform. To uncompress and expand the tar archive, run % tar xvfz basplnx-{version}.{arch}.tgz The installation process involves the following steps: (1) To build kernel module, "basp.o", % make Note that the Make process will automatically build the correct module for different kernel options, e.g. symbol versioning and SMP support. There is NO need to define -DMODVERSIONS in the Makefile. (2) To create device file and to copy files, % make install (3) To update the module reference, % depmod -a (4) To load the driver, % insmod basp (5) Follow the "Configuration and Startup for Other Linux Distribution" section to set up the teams. 3.2 Install - RPM package ------------------------- (1) To install the RPM source package, run % rpm -i basplnx-{version}.src.{arch}.rpm (2) Change directory to the RPM path and build the binary driver for the kernel % cd /usr/src/redhat % rpm -bb SPECS/basplnx.spec Note that the RPM path is different for different Linux distributions. For those distributions with RPM v4.1 or later, use "rpmbuild" comannd instead of "rpm". (3) Install the newly built package % rpm -i RPMS/i386/basplnx-{version}.{arch}.rpm The driver and other required files will be automatically installed. (4) To load the driver, % insmod basp (5) Follow the Configuration section to set up the teams. 4.0 Configuration Information _______________________________ 4.1 Configuration - Red Hat & United Linux based Distribution ------------------------------------------------------------ The BASP distribution includes a utility program and several scripts for team configuration. Most of the steps are only required to be performed after the first time installation. Step 2 "Modify the configuration script" should be performed whenever there is any change to the team configuration. Since Linux distributions do not automatically load drivers for network devices unless the device is configured with an IP address, users must manually configure a network-script file for all physical adapters that will be team members. Network script files are located under /etc/sysconfig/network-scripts (Red Hat), or /etc/sysconfig/network (United Linux 1.x). The file name must be prefixed with "ifcfg-" then the physical adapter alias. For interface eth0, you would create a file with the name ifcfg-eth0, then add the below content. Example: DEVICE=eth0 BOOTPROTO=static ONBOOT=yes For users of other Linux distributions, follow instructions in the "BASP Configuration and Startup for Other Linux Distributions" section. The configuration process involves the following steps: Copy a configuration script from the "/etc/basp/samples" directory to the "/etc/basp" directory. Note that the configuration script name must be prefixed with "team-". Modify the configuration script to: (a) change the team type (b) add/delete the physical network interfaces (c) add/delete the virtual network interfaces (d) assign IP address to each virtual network interface. The syntax of the configuration script can be found in the BASP Configuration Scripts for Red Hat Distributions section or in the /etc/basp/sample/team-sample script file itself. Note that when configuring Teaming, at least one Primary Adapter is required. Manually start the team for the first time: % /etc/init.d/basp start NOTE - This step is only required for the first time installation. The team configuration will be automatically started on subsequent reboots. Note that if not all the virtual network interfaces are configured with an IP address, there will be an error message in starting the BASP team. When this happens, repeat Step (2) to configure an IP address for all the virtual network interfaces. Note-1) Forming multiple teams is possible by copying the sample files into "/etc/basp/team-" and modifying this file as described in the sample file. Note-2) In order to create more that one virtual interface (VLAN) for each team, refer to the respective description section in the sample files. 4.2 Configuration - Suse Distribution ------------------------------------ The BASP distribution includes an utility program and several scripts for team configuration. Following steps for Suse Linux distributions only. Most of the steps are only required to be performed after the first time installation. Step 2 "Modify the configuration script" should be performed whenever there is any change to the team configuration. Since Suse distribution does not automatically load driver for the network device unless the device is configured with IP address, users must manually use "yast2" or configure /etc/rc.config file to ensure proper network drivers loaded during init time. For users of other Linux distributions, follow instructions in the "baspcfg" section. The configuration process involves the following steps: (1) Copy a configuration script from "/etc/basp/samples" directory to "/etc/basp" directory. Note the configuration script must be prefixed with "team-". (2) Modify the configuration script to (a) change the team type (b) add/delete physical network interfaces (c) add/delete virtual network interfaces (d) assign IP address to each virtual network interface The syntax of the configuration script can be found below. Note that when configuring Teaming, at lease one Primary Adapter is required. (3) Manually start the team for the first time, % /etc/init.d/basp start Note that this step is only required for the first time installation. The team configuration will be automatically started on subsequent reboots. Note that if not all the virtual network interfaces are configured with IP address, there will be an error message in starting the BASP team. When this happens, repeat step (2) to configure IP address for all the virtual network interfaces. Note-1) Forming multiple teams is possible by copying the sample files into "/etc/basp/team-" and modifying this file as described in the sample file. Note-2) In order to create more that one virtual interface (VLAN) for each team, refer to the respective description section in the sample files. 4.3 Configuration and Startup for Other Linux Distribution ---------------------------------------------------------- "baspcfg" is a command line tool to configure the BASP teams, add/remove NICs and add/remove virtual devices. This tool, may be used to configure your own startup script files. Please read your distribution specific documentation for more information on your distributions's startup procedure. Following is the usage of this tool: baspcfg v6.0.1 - Broadcom Advanced Server Program Configuration Utility Copyright (c) 2000-2003 Broadcom Corporation. All rights reserved. usage: baspcfg commands: addteam -- create a team delteam -- delete a team addva [macaddr] -- add a virtual adapter to a team delva -- del a virtual adapter from a team bind -- bind a physical adapter to a team unbind -- unbind a physical adapter from a team show [tid] -- display team configurations failback -- manual failback on SLB (Auto-Fallback Disable) team only. where tid -- An unique ID for each team, starting from 0 type -- Team type: 0=SLB, 1=FEC/GEC, 2=802.3ad, 3=SLB (Auto-Fallback Disable) tname -- ASCII string of the team vlan_id -- VLAN ID: from 1 to 4094, 0=untagged or no VLAN vname -- ASCII string of the virtual device macaddr -- MAC address (optional), e.g. 00:10:18:00:11:44 role -- Role of the physical device: 0=primary, 1=hot-standby device -- ASCII string of the physical device, e.g. eth0 Following sample startup script should be used to start the BASP after first time installation and configuration, or in the subsequent reboots. #!/bin/bash # load basp module insmod basp # create new team baspcfg addteam 0 0 team-one # bind physical interfaces / two primary one backup baspcfg bind 0 0 eth0 baspcfg bind 0 0 eth1 baspcfg bind 0 1 eth2 # create the virtual interface baspcfg addva 0 0 sw0 # bind ip address to virtual interface and initialize ifconfig sw0 192.168.0.1 up Note: Baspcfg command can only be executed in Super User mode only, configure with other login mode will yield "Permission Denied. Must be root to manage BASP" error message. When configuring Teaming, at lease one Primary Adapter is required. 4.4 Startup scripts - Red Hat distributions ------------------------------------------- (1) basp This script is intended to be installed in /etc/init.d directory. After copying the script, run "chkconfig --add basp". This script will be executed at runlevel 2, 3, 4 and 5. When "basp" is run, it will search the /etc/basp directory to list all the files with "team-" prefix, and then it will invoke "baspteam" script to add or delete the teams. It is normal that each "team-*" file in /etc/basp represents 1 team. (2) baspteam This script is called by "basp" script to add or delete a team. To install, create "/etc/basp" directory and copy this script over. To manually add a team: % baspteam team-sample add To delete a team: % baspteam team-sample del Note that "team-sample" is the configuration script. (3) team-sample This script contains a SLB team configuration with 3 NICs: eth0, eth1 and eth2. The team name is "TeamSample". All 3 NICs are primary. One virtual interface is also created for this team and the name of the virtual interface is "sw0". "sw0" is the device that "ifconfig" should be run against to set up the IP address. VLANs are not enabled in this script. This script and "team-gec" are intended to be customized. Refer to the configuration scripts section for details. This script should be copied to /etc/basp directory and retain the "team-" prefix. (4) team-gec This configuration script creates a GEC team with 3 network interfaces, eth0, eth1 and eth2. The team name is "TeamGEC". All 3 NICs are primary. One virtual interface is added to the team with the name "sw0" and VLANs are not enabled. This script and "team-sample" are intended to be customized. Refer to the configuration scripts section for details. This script should be copied to /etc/basp directory and retain the "team-" prefix. 4.5 Configuration scripts - Red Hat and Suse -------------------------------------------- Both "team-sample" and "team-gec" are configuration scripts that follow the same syntax, as following TEAM_ID: this number uniquely identifies a team. TEAM_TYPE: 0 = SLB, 1 = Generic Trunking/GEC/FEC, 2 = 802.3ad, 3 = SLB (Auto-Fallback Disable) TEAM_NAME: ascii name of the team TEAM_PAx_NAME: ascii name of the physical interface x, where x can be 0 to 7. TEAM_PAx_ROLE: role of the physical interface x 0 = Primary, 1 = Hot-standby. This field must be 0 for Generic Trunking/GEC/FEC team. TEAM_VAx_NAME: ascii name of the virtual interface x, where x can be 0 to 63 TEAM_VAx_VLAN: 802.1Q VLAN ID of the virtual interface x. For untagged virtual interface, i.e. without VLAN enable, set it to 0. The valid VLAN ID can be 0 to 4094. TEAM_VAx_IP: IP address of the virtual interface x. The format should be aa.bb.cc.dd. TEAM_VAx_NETMASK: Subnet mask of the virtual interface x. The format should mm.nn.oo.pp. Note: Teaming scripts are intended for Red Hat distributions ONLY, use with other Linux distribution will causes ERROR. 4.6 Broadcom NICE patches ------------------------- Also included in this release are network device drivers patched with Broadcom NICE support. These drivers are originally taken from the Linux 2.4.16 kernel distribution. To install patched drivers: (1) Copy the Broadcom NICE header file, "nicext.h", to the appropriate Linux kernel include directory, e.g. % cp /usr/src/nice-2.4.16/nicext.h /usr/src/linux/include/linux (2) Rename the original network device driver under the Linux kernel source tree, "/usr/src/linux/drivers/net". (3) Copy the patched drivers to the Linux kernel network driver source directory, i.e. "/usr/src/linux/drivers/net". (4) Follow the kernel rebuild instructions to configure kernel support for these drivers, i.e. % cd /usr/src/linux % make config (5) If the patched drivers are configured into the kernel, goto step (7). If the patched drivers are configured as modules, goto step (6). (6) In the case of supporting only the module version of these drivers, it is possible to simply run the following to compile patched drivers and to install them into the proper module directory: % make modules % make modules_install There is no need to compile the complete kernel. Goto step (8). (7) Rebuild the kernel to compile these patched drivers % make clean % make dep % make (8) Either reboot the system or unload/load the patched modules. Run configuration scripts to test the patch. 4.7 Uninstall - RPM package --------------------------- To uninstall RPM package, % rpm -e basplnx and to reboot the system, % reboot 4.8 Removal of physical interface in Generic Trunking and 802.3ad mode ---------------------------------------------------------------------- In Generic Trunking and 802.3 ad mode, all the physical and virtual interfaces belonging to a team have the same MAC address. This MAC address is the same address as that of the first physical interface bounded to the team. In the case that this first physical interface is removed dynamically from the team using "baspcfg" tool and bounded to the protocol directly, this could lead to a duplicate MAC address problem on the network. Note that if the removed physical interface does not participate in any traffic, there will not be any problem. To properly remove a physical interface, follow the steps listed below: (1) Backup the original team configuration script % cp /etc/basp/team-gec /etc/basp/backup-gec Note-1) "team-gec" is the name of the configuration script. Note-2) "backup-gec" is the name of the backup script. The name of the backup script must NOT be prefixed with "team-". (2) Modify the team configuration script to remove the physical interface (3) Stop the running team % /etc/basp/baspif /etc/basp/backup-gec stop % /etc/basp/baspteam /etc/basp/backup-gec del (4) Re-start the team % /etc/basp/baspteam /etc/basp/team-gec add % /etc/basp/baspif /etc/basp/team-gec start 5.0 WEB Sites and Support Phone Number ________________________________________ IBM Support Web Site: http://www.pc.ibm.com/support IBM Marketing Netfinity Web Site: http://www.pc.ibm.com/us/eserver/xseries If you have any questions about this update, or problems applying the update go to the following Help Center World Telephone Numbers URL: http://www.ibm.com/planetwide. 6.0 Trademarks and Notices ____________________________ The following terms are trademarks of the IBM Corporation in the United States or other countries or both: IBM Netfinity eServer xSeries Broadcom and NetXtreme are registered trademarks of Broadcom Corporation. Other company, product, and service names may be trademarks or service marks of others. 7.0 License and Disclaimer ___________________________ Copyright (c) 2000-2004, Broadcom Corporation Copyright (c) 2001-2004, IBM Corporation All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of IBM Corporation nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.