System Management Guide: Communications and Networks

EtherChannel

EtherChannel is a network port aggregation technology that allows several Ethernet adapters to be aggregated together to form a virtual Ethernet adapter. The adapters that belong to an EtherChannel are cabled to the same EtherChannel-enabled network switch, which must be manually configured to identify the ports that belong to the EtherChannel.

The system sees the EtherChannel as any other Ethernet adapter. For this reason, IP is configured over an EtherChannel adapter as over any Ethernet adapter. Furthermore, all the adapters in the EtherChannel are automatically configured with the same hardware (MAC) address, so they are treated by remote systems as if they were one adapter.

EtherChannel's main benefit is that its interface has the aggregated network bandwidth of all its adapters. Traffic is distributed across the adapters in either the standard way (where the adapter over which the packets are sent is chosen depending on the destination address) or on a round-robin basis (where packets are sent evenly across all adapters). If an adapter fails, the packets are automatically sent on the next available adapter in the EtherChannel without disruption to existing user connections. Once the adapter failure has been corrected, the adapter can be used for network traffic once again.

In AIX, users can configure multiple EtherChannels per system, but it is required that all the links in one EtherChannel are attached to a single switch. Because the EtherChannel cannot be spread across two switches, the entire EtherChannel is lost if the switch is unplugged or fails. To solve this problem, a new backup option added in AIX 5.2 keeps the service running when the main EtherChannel fails. The backup and EtherChannel adapters should be attached to different network switches. In the event that all of the adapters in the EtherChannel fail, the IP and MAC addresses will be automatically moved to the backup adapter. When any link in the EtherChannel is restored, the service is moved back to the EtherChannel.

Network Interface Backup, a mode of operation for EtherChannel, protects against a single point of Ethernet network failure. In Network Interface Backup mode, only one adapter at a time is actively used for network traffic. The EtherChannel tests the currently-active adapter and, optionally, the network path to a user-specified node. When a failure is detected, the MAC and IP addresses are moved to the next adapter, which will be used until it fails. Network Interface Backup provides rapid detection and failover with no disruption to user connections. Network Interface Backup was originally implemented as a mode in the EtherChannel SMIT menu. In AIX 5.2, the backup adapter provides the equivalent function, so the mode was eliminated from the SMIT menu. To use network interface backup in AIX 5.2, see Configure Network Interface Backup.

Configuring EtherChannel

The following procedure will help you set up an EtherChannel.

Considerations

• You can have anywhere from two to eight Ethernet adapters per EtherChannel.
• You can configure multiple EtherChannels on a single system, but remember that each EtherChannel constitutes an additional Ethernet interface. The no command option, ifsize, may need to be increased to include not only the Ethernet interfaces for each adapter, but also any EtherChannels that are configured. The default ifsize is eight.
• You can use any supported Ethernet adapter in EtherChannel. However, the Ethernet adapters must be connected to a switch that supports EtherChannel. See the documentation that came with your switch to determine if it supports EtherChannel.
• All adapters in the EtherChannel should be configured for the same speed (10 Mbps, for example) and must be configured with the same mode (full duplex, for example).
• The adapters that you plan to use for your EtherChannel must not have an IP address configured on it before you start this procedure. Use the ifconfig command to unconfigure your adapters. For example, ifconfig en5 detach would unconfigure adapter en5.

Configure an EtherChannel

1. With root authority, type smit etherchannel at the command line.
2. Select Add an EtherChannel from the list and press Enter.
3. Select the primary Ethernet adapters that you want on your EtherChannel and press Enter. If you are planning to use EtherChannel backup, do not select the adapter that you plan to use for the backup at this point. The EtherChannel backup option was added in AIX 5.2.
   Note
   The Available Network Adapters displays all Ethernet adapters. If you select an Ethernet adapter that is already being used (has an interface defined), you will get an error message. You first need to detach these interfaces if you want to use them.
4. Enter the information in the fields according to the following guidelines:
   • EtherChannel Adapters: You should see all primary adapters that you are using in your EtherChannel. You selected these adapters in the previous step.
   • Enable Alternate EtherChannel Address: This field is optional. Setting this to yes will enable you to specify a MAC address that you want the EtherChannel to use. If you set this option to no, the EtherChannel will use the MAC address of the first adapter.
   • Alternate EtherChannel Address: If you set Enable Alternate EtherChannel Address to yes, specify the MAC address that you want to use here. The address you specify must start with 0x and be a 12-digit hexadecimal address.
   • Enable Gigabit Ethernet Jumbo Frames: This field is optional. In order to use this, your switch must support jumbo frames. This will only work with Standard Ethernet, not IEEE 802.3. Set this to yes if you want to enable it.
   • Mode: You can choose from the following modes:
     • Standard: In this mode the EtherChannel uses the destination IP address to choose which adapter it will send the packets out on. The EtherChannel divides the last byte of the packet's destination IP address by the number of adapters in the EtherChannel and uses the remainder (using the "modulus" operator) to identify the outgoing link. For instance, if the destination IP is 10.10.10.1, and there are 2 adapters in the EtherChannel, (1 % 2) = 1, so the second adapter is used (the adapters are numbered starting from 0). The adapters are numbered in the order they are listed in the SMIT menu. For non-IP traffic (such as ARP), the last byte of the destination MAC address is used to do the calculation. This mode will guarantee packets are sent out over the EtherChannel in the order they were given to it, but it may not make full use of the bandwidth. This is the default operation mode.
     • Round Robin: In this mode the EtherChannel will rotate through the adapters, giving each adapter one packet before repeating. The packets may be sent out in a slightly different order than they were given to the EtherChannel, but it will make the best use of its bandwidth.
     • Network Interface Backup: This option is available in AIX 5.1 and AIX 4.3.3. In this mode, the channel will activate only one adapter at a time. The intention is that the adapters are plugged into different Ethernet switches, each of which is capable of getting to any other machine on the subnet or network. When a problem is detected either with the direct connection (or optionally through the inability to ping a machine), the EtherChannel will deactivate the current adapter and activate a backup adapter. This mode is the only one that makes use of the Internet Address to Ping, Number of Retries, and Retry Timeout fields.

       Network Interface Backup Mode does not exist as an explicit mode in AIX 5.2. To enable Network Interface Backup Mode in AIX 5.2, you must configure one adapter in the main EtherChannel and a backup adapter. For more information, see Configure Network Interface Backup.

   • Backup Adapter: This field is optional. Enter the adapter that you want to use as your EtherChannel backup. EtherChannel backup is available in AIX 5.2.
   • Internet Address to Ping: This field is optional and is only available if you are running Network Interface Backup mode. The EtherChannel will ping the IP address that you specify here. If the EtherChannel is unable to ping this address for the Number of Retries times in Retry Timeout intervals, the EtherChannel will switch adapters.
   • Number of Retries: Enter the number of ping response failures that are allowed before the EtherChannel switches adapters. The default is three. This field is optional and valid only if you have set an Internet Address to Ping.
   • Retry Timeout: Enter the number of seconds between the times when the EtherChannel will ping the Internet Address to Ping. The default is one second. This field is optional and valid only is you have set an Internet Address to Ping.
5. Press Enter after changing the desired fields to create the EtherChannel.
6. Configure IP over the newly created EtherChannel device by typing smit chinet at the command line.
7. Select your new EtherChannel interface from the list.
8. Fill in all the required fields and press Enter.

Configure Network Interface Backup

Network Interface Backup protects against a single point of network failure by providing failure detection and failover with no disruption to user connections. When operating in this mode, only one adapter is active at any given time. If the active adapter fails, the next adapter in the EtherChannel will be used for all traffic. When operating in Network Interface Backup mode, it is not necessary to connect to an EtherChannel-enabled switch.

The Network Interface Backup setup is most effective when the adapters are connected to different network switches, as this provides greater redundancy than connecting all adapters to one switch. When connecting to different switches, make sure there is a connection between the switches. This provides failover capabilities from one adapter to another by ensuring that there is always a route to the currently-active adapter.

In releases previous to AIX 5.2, Network Interface Backup mode was implemented as an explicit mode of operation in the EtherChannel SMIT menu. In AIX 5.2, however, the backup adapter functionality provides the equivalent behavior, so the mode was eliminated from the SMIT menu.

AIX 5.2 additionally provides priority, meaning that the adapter configured in the primary EtherChannel will be used over the backup adapter. As long as the primary adapter is functional, it will be used. This contrasts from the behavior of Network Interface Backup mode, where the backup adapter was used until it also failed, regardless of whether the primary adapter had already recovered.

While operating in Interface Backup Mode, it is also possible to configure the EtherChannel to detect link failure and network unreachability. To do this, specify the IP address of a remote host where connectivity should always be present. The EtherChannel will periodically ping this host to determine whether there is still a network path to it. If a specified number of ping attempts go unanswered, the EtherChannel will fail over to the next backup adapter in the hope that there is a network path to the remote host through the next adapter. In this setup, not only should every adapter be connected to a different switch, but each switch should also have a different route to the host that is pinged.

This ping feature is only available in Network Interface Backup mode. However, in AIX 5.2, if the ping feature is enabled and a failover due to ping failure has occurred, the EtherChannel will not fail back to the primary adapter. The backup adapter will be the active channel as long as it is working because there is no way to know when the route to the pinged host will become reachable from the primary adapter. If a failure is detected while the backup adapter is active (that is, if either the ping attempts fail from the backup adapter or if the backup adapter itself fails), the EtherChannel will then failover to the primary adapter. If the failover occurred because the primary adapter failed, the EtherChannel will then come back to the primary adapter as soon it has come back up.

To configure Network Interface Backup in AIX 5.2, see Configure Network Interface Backup in AIX 5.2. To configure Network Interface Backup in previous versions of AIX, see Configure Network Interface Backup in previous AIX versions

Configure Network Interface Backup in AIX 5.2

1. With root authority, type smit etherchannel on the command line.
2. Select Add an EtherChannel from the list and press Enter.
3. Select the primary Ethernet adapter and press Enter. This is the adapter that will be used until it fails.
   Note
   The Available Network Adapters displays all Ethernet adapters. If you select an Ethernet adapter that is already being used, you will get an error message and will need to detach this interface before you can use it. See the ifconfig command for information on how to detach an interface.
4. Enter the information in the fields according to the following guidelines:
   • EtherChannel Adapters: You should see the primary adapter you selected in the previous step.
   • Enable Alternate EtherChannel Address: This field is optional. Setting this to yes will enable you to specify a MAC address that you want the EtherChannel to use. If you set this option to no, the EtherChannel will use the MAC address of the primary adapter.
   • Alternate EtherChannel Address: If you set Enable Alternate EtherChannel Address to yes, specify the MAC address that you want to use here. The address you specify must start with 0x and be a 12-digit hexadecimal address.
   • Enable Gigabit Ethernet Jumbo Frames: This field is optional. In order to use this, your switch must support jumbo frames. This will only work with Standard Ethernet, not IEEE 802.3. Set this to yes if you want to use it.
   • Mode: It is irrelevant which mode of operation you select because there is only one adapter in the main EtherChannel. All packets will be sent over that adapter until it fails. There is no netif_backup mode because that mode can be emulated using a backup adapter.
   • Backup Adapter: Enter the adapter that you want to be your backup adapter. After a failover, this adapter will be used until the primary adapter recovers. It is recommended to use the preferred adapter as the primary adapter.
   • Internet Address to Ping: The field is optional. The EtherChannel will ping the IP address that you specify here. If the EtherChannel is unable to ping this address for Number of Retries time in Retry Timeout intervals, the EtherChannel will switch adapters.
   • Number of Retries: Enter the number of ping response failures that are allowed before the EtherChannel switches adapters. The default is three. This field is optional and valid only if you have set an Internet Address to Ping.
   • Retry Timeout: Enter the number of seconds between the times when the EtherChannel will ping the Internet Address to Ping. The default is one second. This field is optional and valid only if you have set an Internet Address to Ping.
5. Press Enter after changing the desired fields to create the EtherChannel.
6. Configure IP over the new interface by typing smit chinet at the command line.
7. Select your new EtherChannel interface from the list.
8. Fill in all the required fields and press Enter.

Configure Network Interface Backup in previous AIX versions

1. With root authority, type smit etherchannel on the command line.
2. Select Add an Etherchannel and press Enter.
3. Select the adapters that you want to include in the EtherChannel, both primary and secondary.
   Note
   The Available Network Adapters displays all Ethernet adapters. If you select an Ethernet adapter that is already being used, you will get an error message. You first need to detach that interface by using the ifconfig command.
4. Enter the information in the fields according to the following guidelines:
   • Etherchannel adapters: You should see the adapters that you selected in the previous step.
   • Enable ALTERNATE ETHERCHANNEL address: This field is optional. Setting this to yes will enable you to specify a MAC address that you want the EtherChannel to use. If you set this option to no, the EtherChannel will use the MAC address of the first adapter specified.
   • ALTERNATE ETHERCHANNEL address: If you set Enable ALTERNATE ETHERCHANNEL address to yes, specify the MAC address that you want the EtherChannel to use here. The address you specify must start with 0x and be a 12-digit hexadecimal value.
   • Mode: Select netif_backup.
   • Enable GIGABIT ETHERNET JUMBO frames: This field is optional. In order to use this, your switch must support jumbo frames. This will only work on Standard Ethernet, not IEEE 802.3. Set this to yes if you want to enable it.
   • Internet Address to Ping: This field is optional. The EtherChannel will ping the IP address you specify here. If the EtherChannel is unable to ping this address for Number of Retries times in Retry Timeout intervals, the EtherChannel will switch adapters.
   • Number of Retries: Enter the number of ping response failures that are allowed before the EtherChannel switches adapters. The default is three. This field is optional and valid only if you set an Internet Address to Ping.
   • Retry Timeout: Enter the number of seconds between the times when the EtherChannel will ping the Internet Address to Ping. The default is one second. This field is optional and valid only if you have set and Internet Address to Ping.
5. Press Enter after changing the desired fields to create the EtherChannel.
6. Configure IP over the new interface by typing smit chinet at the command line.
7. Select your new EtherChannel interface from the list.
8. Fill in all required fields and press Enter.

Managing an EtherChannel

This section will tell you how to perform the following tasks:

• List EtherChannels
• Change the Alternate Address
• Add, remove, or change adapters in an EtherChannel
• Remove an EtherChannel
• Configure or remove a backup adapter on an existing EtherChannel

List EtherChannels

1. On the command line, type smit etherchannel.
2. Select List All EtherChannels and press Enter.

Change the Alternate Address

This enables you to specify a MAC address for your EtherChannel.

1. Detach the EtherChannel by typing ifconfig interface detach, where interface is your EtherChannel's interface.
2. On the command line, type smit etherchannel.
3. Select Change / Show Characteristics of an EtherChannel and press Enter.
4. If you have multiple EtherChannels, select the EtherChannel for which you want to create an alternate address.
5. Change the value in Enable Alternate EtherChannel Address to yes.
6. Enter the alternate address in the Alternate EtherChannel Address field. The address must start with 0x and be a 12-digit hexadecimal address.
7. Press Enter to complete the process.

Add, remove, or change adapters in an EtherChannel

1. Detach the EtherChannel by typing ifconfig interface detach, where interface is your EtherChannel's interface.
2. On the command line type, smit etherchannel.
3. Select Change / Show Characteristics of an EtherChannel and press Enter.
4. Select the EtherChannel that you want to modify.
5. Select the primary adapters that you want in your EtherChannel and press Enter. If you are using EtherChannel backup, do not select that adapter here.
6. Fill in the necessary fields and press Enter.

Remove an EtherChannel

1. Detach the EtherChannel by typing ifconfig interface detach, where interface is your EtherChannel's interface.
2. On the command line type smit etherchannel.
3. Select Remove an EtherChannel and press Enter.
4. Select the EtherChannel that you want to remove and press Enter.

Configure or remove a backup adapter on an existing EtherChannel

The following procedure configures or removes a backup adapter on an EtherChannel. This option is available only in AIX 5.2.

1. Detach the EtherChannel by typing ifconfig interface detach, where interface is your EtherChannel's interface.
2. On the command line, type smit etherchannel.
3. Select Change / Show Characteristics of an EtherChannel.
4. Select the EtherChannel that you are adding or modifying the backup adapter on.
5. Enter the adapter that you want to use as your backup adapter in the Backup Adapter field, or select NONE if you wish to stop using the backup adapter.

Troubleshooting EtherChannel

If you are having trouble with your EtherChannel, you can use tcpdump and iptrace to troubleshoot the EtherChannel. The trace hook id for the transmission packets is 2FA and for other events is 2FB. You cannot trace receive packets on the EtherChannel as a whole, but you can trace each adapter's receive trace hooks.

You can also use the entstat command to get the aggregate statistics of all the adapters in the EtherChannel. For example, entstat ent7 will display the aggregate statistics of ent7. Adding the -d flag will also display the statistics of each adapter individually. For example, typing entstat -d ent7 will show you the aggregate statistics of the EtherChannel as well as the statistics of each individual adapter in the EtherChannel.

If you notice that the failover time when you are using network interface backup mode or EtherChannel backup is much greater than five seconds, you should verify that your switch is not running the Spanning Tree Protocol (STP). When the switch detects a change in its mapping of switch port to MAC address, it runs the spanning tree algorithm to see if there are any loops in the network, and this may take some time. Network Interface Backup and EtherChannel backup may cause a change in the port to MAC address mapping, so disabling the algorithm should make the failover faster.

Also, note that switch ports have a forwarding delay counter that determines how soon after it is initialized each port should begin forwarding or sending packets. For this reason, when the main channel is re-enabled, you may notice that there is a delay of a few seconds before the connection is re-established, whereas the failover to the backup adapter is much faster. You might want to check the forwarding delay counter configured in your switch and make it as small as possible so that coming back to the main channel occurs as fast as possible.

Specifically, for the EtherChannel backup function to work correctly, the forwarding delay counter must not be more than 10 seconds, or coming back to the main EtherChannel might not work correctly. Setting the forwarding delay counter to the lowest value allowed by the switch is recommended.

If you find that adapter failures are not triggering failovers (in other words, if an adapter that is down is still being used by an EtherChannel to send packets), you might want to check to see if your adapter card needs to have link polling enabled to detect link failure. Some adapters cannot automatically detect their link status (whether they are up or down). To detect said condition, these adapters must enable a link polling mechanism, whereby a timer is started that periodically verifies the status of the link. This link polling is disabled by default because it decreases performance. For EtherChannel to work correctly with these adapters, however, the link polling mechanism must be enabled on each adapter before the EtherChannel is created.

Adapters that have a link polling mechanism have an ODM attribute called poll_link, which must be set to yes for the link polling to be enabled. Before creating the EtherChannel, use the following command on every adapter to be included in the channel:

chdev -l ent? -a poll_link=yes

Likewise, for the jumbo frames option to work properly, aside from enabling the use_jumbo_frame attribute on the EtherChannel, you must also enable jumbo frames on each adapter before creating the EtherChannel using the following command:

chdev -l ent? -a jumbo_frames=yes

Remote dump is not supported over an EtherChannel.