TCP/IP Problem Determination

This section contains information about diagnosing common problems in a Transmission Control Protocol/Internet Protocol (TCP/IP) network environment.

The netstat command is a good tool to use to help determine in which area the problem is. Once you have isolated the problem to an area, you can use more sophisticated tools to proceed. For example, you might use the netstat -i and netstat -v to determine if you have a problem with a particular hardware interface, and then run diagnostics to further isolate the problem. Or, if the netstat -s command shows that there are protocol errors, you could then use the trpt or iptrace commands.

The topics discussed in this section are:

• Communication Problems
• Name Resolution Problems
• Routing Problems
• Problems with System Resource Controller (SRC) Support
• telnet or rlogin Problems
• Configuration Problems
• Common Problems with Network Interfaces
• Problems with Packet Delivery
• Problems with Dynamic Host Configuration Protocol (DHCP)

Communication Problems

If you cannot communicate with a host on your network:

• Try to contact the host, using the ping command. Run the ping command on the local host to verify that the local interface to the network is up and running.
• Try to resolve the name of the host, using the host command. If the name does not resolve, you have a name resolution problem. See "Name Resolution Problems" for more information.

If the name resolves and you are trying to contact a host on another network, you may have a routing problem. See "Routing Problems" for more information.

• If your network is a token-ring network, check to see if the target host is on another ring. If so, the allcast field is probably set incorrectly. Use the Web-based System Manager fast path, wsm network, or the System Management Interface Tool (SMIT) fast path smit chinet to access the Network Interfaces menu. Then, set the Confine Broadcast to Local Ring field to no in the token-ring dialog.
• If there are a large number of Address Resolution Protocol (ARP) packets on your network, verify that your subnet mask is set correctly. This condition is known as a broadcast storm and may affect your system's performance.

Name Resolution Problems

Resolver routines on hosts running TCP/IP attempt to resolve names, using the following sources in the order listed:

1. DOMAIN name server (named)
2. Network Information Service (NIS)
3. Local /etc/hosts file.

Client Host

If you cannot get a host name resolved, and you are using flat name resolution (using the /etc/hosts file), verify that the host name and correct Internet Protocol (IP) address information is in the /etc/hosts file.

If you cannot get a host name resolved, and you are using a name server:

1. Verify that you have a resolv.conf file specifying the domain name and Internet address of a name server.
2. Verify that the local name server is up by issuing the ping command with the IP address of the name server (found in the local resolv.conf file).
3. If the local name server is up, verify that the named daemon on your local name server is active by issuing the ping command with the IP address of the name server (found in the local resolv.conf file).
4. If you are running the syslogd, there could be error messages logged. The output for these messages is defined in the /etc/syslog.conf file.

If these steps do not identify the problem, start looking at the name server host.

Name Server Host

If you cannot get a host name resolved:

1. Verify that the named daemon is active by issuing the following command:

   lssrc -s named

2. Verify that the address of the target host exists and is correct in the name server's database. Send a SIGINT signal to the named daemon to dump the database and cache to the file /var/tmp/named_dump.db. Verify that the address you are trying to resolve is there and is correct.

   Add or correct name-to-address resolution information in the named hosts data file for the primary name server of the domain. Then issue the following SRC command to reread the data files:

   refresh -s named

3. Verify that the name resolution requests are being processed. To do this, enter the named daemon from the command line and specify a debugging level. Valid debug levels are 1 through 9. The higher the level, the more information the debug mechanism logs.

   startsrc -s named -a "-d DebugLevel"

4. Check for configuration problems in the named data files. For more information, see "Configuring Name Servers", the "DOMAIN Data File Format," "DOMAIN Reverse Data File Format," "DOMAIN Cache File Format," and the "DOMAIN Local Data File Format" in the AIX Version 4.3 Files Reference.

   Note: A common error is the incorrect use of the . (period) and the @ (at sign) in the DOMAIN data files.

If external users cannot reach your domains:

• Make sure that all your nonprimary name servers (secondary, cache-only) have equal time-to-live (TTL) information in the DOMAIN data files.

If external resolvers query your servers constantly:

• Make sure your servers are distributing DOMAIN data files with reasonable TTL values. If the TTL is zero or another small value, the data you transfer times out very quickly. Set the minimum value in your start of authority (SOA) records to a week or more to solve this problem.

Routing Problems

If you cannot reach a destination host, consider the following situations:

• If you receive a Network Unreachable error message, make sure that a route to the gateway host has been defined and is correct. Check this by using the netstat -r command to list kernel routing tables.
• If you receive a No route to host error message, verify that the local network interface is up by issuing the ifconfig interface_name command. The output should indicate that the interface is up. Use the ping command to try and reach another host on your network.
• If you receive a Connection timed out error message:
  • Verify that the local gateway is up using the ping command with the name or Internet address of the gateway.
  • Make sure that a route to the gateway host has been defined and is correct. Check this by using the netstat -r command to list kernel routing tables.
  • Make sure the host you want to communicate with has a routing table entry back to your machine.
• If you are using static routing, make sure that a route to the target host and gateway host has been defined. Check this by using the netstat -r command to list kernel routing tables.

  Note: Make sure the host you want to communicate with has a routing table entry to your machine.

• If you are using dynamic routing, verify that the gateway is listed and correct in the kernel routing tables by issuing the netstat -r command.
• If the gateway host is using the Routing Information Protocol (RIP) with the routed daemon, make sure that a static route to the target host is set up in the /etc/gateways file.

  Note: You need to do this only if the routing daemon cannot identify the route to a distant host through queries to other gateways.

• If the gateway host is using the RIP with the gated daemon, make sure that a static route to the target host is set up in the gated.conf file.
• If you are using dynamic routing with the routed daemon:
  • If routed cannot identify the route through queries (for example, if the target host is not running the RIP, check the /etc/gateways file to verify that a route to the target host is defined.
  • Make sure that gateways responsible for forwarding packets to the host are up and running the RIP. Otherwise, you'll need to define a static route.
  • Run the routed daemon using the debug option to log such information as bad packets received. Invoke the daemon from the command line using the following command:

    startsrc -s routed -a "-d"

  • Run the routed daemon using the -t flag, which causes all packets sent or received to be written to standard output. When routed is run in this mode, it remains under the control of the terminal that started it. Therefore, an interrupt from the controlling terminal kills the daemon.
• If you are using dynamic routing with the gated daemon:
  • Verify that the /etc/gated.conf file is configured correctly and that you are running the correct protocols.
  • Make sure the gateway on the source network is using the same protocol as the gateway on the destination network.
  • Make sure that the machine with which you are trying to communicate has a route back to your host machine.
  • Verify that the gateway names in the gated.conf file correspond to the gateway names listed in the /etc/networks file.
• If you are using the RIP or HELLO protocols, and routes to the destination cannot be identified through routing queries, check the gated.conf file to verify that a route to the target host is defined. You should set static routes under the following conditions:
  • The destination host is not running the same protocol as the source host so cannot exchange routing information.
  • The host must be reached by a distant gateway (a gateway that is on a different autonomous system than the source host). The RIP can be used only among hosts on the same autonomous system.

Other Possibilities

If all else fails, you may want to turn on tracing for your routing daemon (either routed or gated). Use the SRC traceson command from the command line, or send a signal to the daemon to specify different levels of tracing. See the gated daemon or the routed daemon for specifics on sending signals to these daemons.

Problems with SRC Support

• If changes to the /etc/inetd.conf file do not take effect:

  Update the inetd daemon by issuing the refresh -s inetd command or the kill -1 InetdPID command.

• If the startsrc -s [subsystem name] returns the following error message:

  0513-00  The System Resource Controller is not active.

  The System Resource Controller subsystem has not been activated. Issue the srcmstr & command to start SRC, then reissue the startsrc command.

  You may also want to try starting the daemon from the command line without SRC support.

• If the refresh -s [subsystem name] or lssrc -ls [subsystem name] returns the following error message:

  [subsystem name] does not support this option.

  The subsystem does not support the SRC option issued. Check the subsystem documentation to verify options the subsystem supports.

• If the following message is displayed:

  SRC was not found, continuing without SRC support.

  A daemon was invoked directly from the command line instead of using the startsrc command. This is not a problem. However, SRC commands, such as stopsrc and refresh, will not manipulate a subsystem that is invoked directly.

telnet or rlogin Problems

The following explanations may be useful in solving problems with the telnet or rlogin command.

Screen Distortion

If you are having trouble with screen distortion in full-screen applications:

1. Check the TERM environment variable by issuing one of the following commands:

   env

   OR

   echo $TERM

2. Verify that the TERM variable is set to a value that matches the type of terminal display you are using.

telnet Debugging

telnet subcommands that may help in debugging problems include:

Programs Using Extended Curses

Problems with function and arrow keys may arise when using the rlogin and telnet commands with programs using extended curses. Function and arrow keys generate escape sequences, which are split if too little time is allotted for the entire key sequence. Curses waits a specific amount of time to decide whether an Esc indicates the escape key only or the start of a multibyte escape sequence generated by other keys, such as cursor keys, the action key, and function keys.

If no data, or data that is not valid, follows the Esc in the allotted amount of time, curses decides that the Esc is the escape key, and the key sequence is split. The delay resulting from the rlogin or telnet command is network dependent. Sometimes arrow and function keys work and sometimes they do not, depending on the speed of the network to which you are connecting. Setting the ESCDELAY environment variable to a large value (1000 to 1500) effectively solves this problem.

Configuration Problems

Network interfaces are automatically configured during the first system startup after the adapter card is installed. However, you still need to set some initial values for TCP/IP including the host name, the Internet address, and the subnet mask. To do this, you can use the Web-based System Manager fast path, wsm network, or, you can use the SMIT interface in the following ways:

• Use the smit mktcpip fast path to set the initial values for the host name, the Internet address, and the subnet mask.
• Use the smit mktcpip fast path to specify a name server to provide name resolution service. (Note that smit mktcpip configures one network interface only.)
• Use the smit chinet fast path to set other network attributes.

You may also want to set up any static routes the host needs for sending transmitting information, such as a route to the local gateway. Use the Web-based System Manager fast path, wsm network, or the SMIT fast path, smit mkroute, to set these up permanently in the configuration database.

If you are having other problems with your configuration, see the "Configuring a TCP/IP Network Checklist" for more information.

Common Problems with Network Interfaces

Network interfaces are configured automatically during the first system startup after the adapter card is installed. However, there are certain values that must be set in order for TCP/IP to start. These include the host name and Internet address and can be set using the Web-based System Manager fast path, wsm network, or the SMIT fast path, smit mktcpip.

If you choose the SMIT method, use the smit mktcpip fast path to set these values permanently in the configuration database. Use the smit chinet and smit hostname fast paths to change them in a running system. The smit mktcpip fast path minimally configures TCP/IP. To add adapters, use the Further Configuration menu, which can be reached with the smit tcpip fast path.

If you have already checked these to verify accuracy and you are still having trouble sending and receiving information, check the following:

• Verify that your network adapter has a network interface by executing the netstat -i command. The output should list an interface, such as tr0, in the Name column. If it does not, create a network interface through Web-based System Manager or by entering the SMIT fast path smit mkinet.
• Verify that IP address for the interface is correct by executing the netstat -i command. The output should list the IP address in the Network column. If it is incorrect, set the IP address through Web-based System Manager or by entering the SMIT fast path smit chinet.
• Check the error log by running the errpt command to ensure that there are no adapter problems.
• Verify that the adapter card is good by running diagnostics.

If these steps do not identify the problem, refer to "Problems with a SLIP Network Interface","Problems with an Ethernet Network Interface", or "Problems with a Token-Ring Network Interface".

Problems with a SLIP Network Interface

In general, the most effective method for debugging problems with a Serial Line Interface Protocol (SLIP) interface is to retrace your configuration, verifying each step. However, you can also:

• Verify that the slattach process is running and using the correct tty port by issuing the ps -ef command. If it is not, run the slattach command. (See "Configuring SLIP over a Modem", or "Configuring SLIP over a Null Modem Cable" for the exact syntax you should use.)
• Verify that the point-to-point addresses are specified correctly by entering the smit chinet fast path.

  Select the SLIP interface. Make sure that the INTERNET ADDRESS and DESTINATION Address fields are correct.

If the modem is not functioning correctly:

• Make sure that the modem was installed properly. See the modem's operator manual.
• Verify that any flow control the modem does is turned off.

If the tty is not functioning properly, verify that the tty's baud rate and modem characteristics are set correctly in the configuration database by entering the smit tty fast path.

Problems with an Ethernet Network Interface

If the network interface has been initialized, the addresses correctly specified, and you have verified that the adapter card is good:

• Verify that you are using a T-connector plugged directly into the inboard/outboard transceiver.
• Make sure you are using Ethernet cable. (Ethernet cable is 50 OHM.)
• Make sure you are using Ethernet terminators. (Ethernet terminators are 50 OHM.)
• Ethernet adapters may be used with either the transceiver that is on the card or with an external transceiver. There is a jumper on the adapter to specify which you are using. Verify that your jumper is set correctly (see your adapter manual for instructions).
• Verify that you are using the proper Ethernet connector type (thin is BNC; thick is DIX). If you change this connector type, use the Web-based System Manager fast path, wsm devices, or the SMIT fast path, smit chgenet, to set the Apply Change to Database Only field. (The field should be checked in Web-based System Manager or set to yes in SMIT.) Reboot the machine to apply the configuration change. (See "Configuring a High-Performance Ethernet Adapter".)

Problems with a Token-Ring Network Interface

If you cannot communicate with some of the machines on your network although the network interface has been initialized, the addresses correctly specified, and you have verified that the adapter card is good:

• Check to see if the hosts with whom you cannot communicate are on a different ring. If they are, use the Web-based System Manager fast path, wsm devices, or the SMIT fast path smit chinet to check the Confine BROADCAST to Local Token-Ring field. The field should not be checked in Web-based System Manager or set to no in SMIT.
• Check to see whether the token-ring adapter is configured to run at the correct ring speed. If it is configured incorrectly, use the Web-based System Manager Network application or SMIT to change the adapter ring speed attribute (see "Configuring a High-Performance Token-Ring Adapter"). When TCP/IP is restarted, the token-ring adapter will have the same ring speed as the rest of the network.

Problems with a Token-Ring/Ethernet Bridge

If you cannot communicate between a token-ring and an Ethernet network, using a bridge, and you have verified that the bridge is functioning properly, the Ethernet adapter may be dropping packets. A machine drops packets if the incoming packet (including headers) is greater than the network adapter's maximum transmission unit (MTU) value. For instance, a 1500-byte packet sent by a token-ring adapter over the bridge collects an 8-byte logical link control (LLC) header, making the total packet size 1508. If the receiving Ethernet adapter's MTU is set to 1500, the packet is dropped.

Check the MTU values of both network adapters. To allow for the eight-byte LLL header, the token-ring adapter attaches to outgoing packets, the MTU value for the token-ring adapter should be set at least eight bytes lower than the MTU value for the Ethernet adapter. For example, the MTU for a token-ring adapter should be set to 1492 to communicate with an Ethernet adapter with an MTU of 1500.

Problems with a Token-Ring/Token-Ring Bridge

When operating through a bridge, the default value of 1500 for the maximum transmission unit (MTU) should be changed to a value that is eight less than the maximum information field (maximum I-frame) advertised by the bridge in the routing control field.

To find the routing control field value, use the iptrace daemon to look at incoming packets. Bits 1, 2, and 3 of Byte 1 are the Largest Frame Bits, which specify the maximum information field that can be transmitted between two communicating stations on a specific route. Refer to the following figure for the format of the routing control field:

Values for the Largest Frame Bits are as follows:

For example, if the maximum I-frame value is 2052 in the routing control field, the MTU size should be set to 2044. This is for token-ring network interfaces only.

Note: When using iptrace, the output file must not be on a Network File System (NFS).

Problems with Packet Delivery

Communicating with a Remote Host

If you cannot communicate with a remote host, try the following:

• Run the ping command on the local host to verify that the local interface to the network is up and running.
• Use the ping command for hosts and gateways that are progressively more hops from the local host to determine the point at which communication fails.

If you are having trouble with packet loss or are experiencing delays in packet delivery, try the following:

• Use the trpt command to trace packets at the socket level.
• Use the iptrace command to trace all protocol layers.

If you cannot communicate between a token-ring and an Ethernet network using a bridge, and you have verified that the bridge is good:

• Check the MTU values of both adapters. The MTU values should be compatible to allow communication. A machine drops packets if the incoming packet (including headers) is greater than the adapter's MTU values. For instance, a 1500-byte packet sent over the bridge collects an 8-byte LLC header, making the total packet size 1508. If the receiving machine's MTU is set to 1500, a packet of 1508 bytes is dropped.

snmpd Response to Queries

If snmpd is not responding to queries and there are no log messages received, the packet may be to large for the kernel User Datagram Protocol (UDP) packet handler. If this is the case, increase the kernel variables, udp_sendspace and udp_recvspace by issuing the following commands:

no -o udp_sendspace=64000
no -o udp_recvspace=64000

The maximum size for a UPD packet is 64K. If your query is larger than 64K, it will be rejected. The packet should be split into smaller packets to avoid this problem.

Problems with Dynamic Host Configuration Protocol (DHCP)

If you cannot get an IP address or other configuration parameters:

• Check to see that you have specified an interface to be configured. This can be done through the Web-based System Manager Network application, by editing the /etc/dhcpcd.ini file, or by using the SMIT fast path smit dhcp.
• Check to see that there is a server on the local network or a relay agent configured to get your requests off the local network.
• Check to see that the dhcpcd program is running. If it is not, use the startsrc -s dhcpcd command.