- The online description of the trpt (TRace Protocol Tcp) command, gives the different states of a TCP connection. I.E. CLOSED Connection is closed. LISTEN Listening for a connection. SYN_SENT Active; have sent SYN. Represents waiting for a matching connection request after having sent a connection request. SYN_RCVD Have sent and received SYN. Represents waiting for a confirming connection request acknowledgment after having both received and sent connection requests. ESTABLISHED Connection established. CLOSE_WAIT Have received FIN; waiting to receive CLOSE. LAST_ACK Have received FIN and CLOSE; awaiting FIN ACK. FIN_WAIT_1 Have closed; sent FIN. CLOSING Closed; exchanged FIN; awaiting FIN. FIN_WAIT_2 Have closed; FIN is acknowledged; awaiting FIN. TIME_WAIT In 2MSL (twice the maximum segment length) quiet wait after close. --------------------------------------------------------------------- Back in June, 1999 (see the entry in my aix.support.center.history file), we were having lots of problems with our web servers. If you do a netstat -An command, you would see the normal 'ESTABLISHED', but you also see a bunch of CLOSE_WAIT, LAST_ACK, FIN_WAIT_1, FIN_WAIT_2 & TIME_WAIT states. The Support Center guy gave me a little education on TCP socket closures. He said there were 2 types of closing, active and passive. Active is when the application (the web server in this case) does the close. Passive is when the client does the close. In both, here is what happens and the states the connection is in; Who Does What Socket State Afterwards ---------------------------- ----------------------- Active: Server Decides to Terminate Connection & Sends a FIN FIN_WAIT_1 Client Sends ACK FIN_WAIT_2 Client Sends a FIN (???) TIME_WAIT Server Sends ACK (???) Closed Passive: Client Decides to Terminate Connection & Sends a FIN. Server sees FIN, sends FIN to application, application (or is it AIX??) sends ACK. CLOSE_WAIT Application does terminating cleanup, then sends a FIN. LAST_ACK Client sends ACK. Closed. The points to note are FIN_WAIT_1 or 2 and TIME_WAIT are Active close states. CLOSE_WAIT & LAST_ACK are passive close states. Also, if sockets are "stuck" in CLOSE_WAIT, it's due to the server application (httpd) not finishing its socket cleanup and sending the last FIN. In other words, it's an application problem. --------------------------------------------------------------------- On an IBM web page that I found when searching on FIN_WAIT (at http://www-1.ibm.com/servlet/support/manager?rt=0&rs=0&org=swg&doc=209DEC0D0607851285256A460053FD0A), I saw more words on FIN_WAIT states. Abstract: "netstat -a" shows many connections with a status of FIN_WAIT_2 Symptom: Note: This symptom does NOT indicate a problem with the IBM HTTP Server. Customer is running the IBM HTTP Server on AIX and is noticing when issuing a netstat -a that many connections have a status of FIN_WAIT_2. This symptom usually occurs on AIX platforms running high traffic websites. Cause: Some clients/browsers never properly close their side of the connection (perhaps a bug in the browser, the clients TCP stack or perhaps the client is dialed in and the dial up link drops before the shutdown completes). There may be other causes, but these are the most common. Impact: If too many FIN_WAIT_2 sessions build up, it can fill up the space allocated for storing connection information and crash the Kernel. Resolution/Workaround: The right way to handle this problem is for the TCP/IP stack to have a fin_wait2 timer that will shutdown sockets stuck in fin_wait2 state. AIX does not have a way to directly tweak the timeout of finwait2 socket but the same thing can be accomplished by tweaking the tcp_keepintvl timer. By default, a socket in TIME_WAIT2 will be cleaned up in 600 seconds (tcp_keepintvl * tcp_keep_alive_retry). tcp_keepintvl is tunable. tcp_keep_alive_retry is not tunable (set to 8). tcp_keepintvl defaults to '150' half second intervals (75 seconds). Times 8 is 600 seconds. You can reduce this 600 second timeout by reducing tcp_keepintvl. The customer should be able to set tcp_keepintvl to as low as perhaps 30 (from 150) which should cause the TIME_WAIT2 sockets to time out in 2 minutes (rather than 10 minutes). How to set the value: First, note what the current setting is (should be 150). no -o tcp_keepintvl login as root no -o tcp_keepintvl=30 For more information on FIN_WAIT_2, please click on the following URL: http://httpd.apache.org/docs/misc/fin_wait_2.html --------------------------------------------------------------------- See the "Internet Encyclopedia" at http://w3search.torolab.ibm.com. RFC's can be viewed at http://cie.bilkent.edu.tr/RFC. and at http://www.faqs.org/rfcs --------------------------------------------------------------------- For a good discussion of the TCP-connection-close states, see RFC 793 at http://cie.bilkent.edu.tr/RFC/index.htm. --------------------------------------------------------------------- To see a good discussion of addressing space, see RFC 1918. For example, it says that 10.0.0.0 - 10.255.255.255 (10/8) 172.16.0.0 - 172.31.255.255 (172.16/12) and 192.168.0.0 - 192.168.255.255 (192/16) are private, non-routable addresses. --------------------------------------------------------------------- Somebody in Southbury mentioned BGP - Autonomous System Number Early 80's See RFC 1107 and/or 1265 --------------------------------------------------------------------- A new command I learned from the Support Center: To trace/see I.P. traffic interactively and in real-time, try tcpdump -i en0 -I instead of iptrace. --------------------------------------------------------------------- To find a machine's I.P. address given only its MAC address, try the trick of sending out a broadcast ping, by pinging the last I.P. address of the subnet, e.g. ping 9.1.15.255 > /dev/null ping 9.1.31.255 > /dev/null or ping 9.1.79.255 > /dev/null for the 3 Almaden domains, and then grep-ing the arp table for the MAC address you're looking for. EG, when looking for 000086073F94, arp -a | grep -i 07:3f:94 --------------------------------------------------------------------- While tracing a NFS problem, this is what I learned about the various protocols used. --------------------------------------------------------------------- sunrpc: Port 111, also known as rpc or portmapper Here's an example portmapper request and reply ====( 98 bytes transmitted on interface en1 )==== 20:33:07.654070784 ETHERNET packet : [ 02:60:8c:2c:97:ec -> 02:60:8c:f5:35:92 ] type 800 (IP) IP header breakdown: < SRC = 192.168.56.252 > (patgate.patent.ibm.com) < DST = 192.168.56.65 > (as0000e1.patent.ibm.com) ip_v=4, ip_hl=20, ip_tos=0, ip_len=84, ip_id=16174, ip_off=0 ip_ttl=30, ip_sum=6add, ip_p = 17 (UDP) UDP header breakdown: [ udp length = 64 | udp checksum = 2120 ] 00000000 34a2a6a5 00000000 00000002 000186a0 |4...............| 00000010 00000002 00000003 00000000 00000000 |................| 00000020 00000000 00000000 000186a5 00000003 |................| 00000030 00000011 00000000 -------- |........ | \---The four words starting here are - The program, x186a5=100005, which if you do a rpcinfo -p, you'll see is mountd. E.G. program vers proto port service 100005 1 udp 4143 mountd 100005 2 udp 4143 mountd 100005 3 udp 4143 mountd - The version. - The protocol, 6=tcp/ip, x11=17=udp - The port, which for a GETPORT request, isn't used. See http://ds.internic.net/rfc/rfc1833.txt. ====( 70 bytes received on interface en1 )==== 20:33:07.658021888 ETHERNET packet : [ 02:60:8c:f5:35:92 -> 02:60:8c:2c:97:ec ] type 800 (IP) IP header breakdown: < SRC = 192.168.56.65 > (as0000e1.patent.ibm.com) < DST = 192.168.56.252 > (patgate.patent.ibm.com) ip_v=4, ip_hl=20, ip_tos=0, ip_len=56, ip_id=51339, ip_off=0 ip_ttl=30, ip_sum=e19b, ip_p = 17 (UDP) UDP header breakdown: [ udp length = 36 | udp checksum = 1e8b ] 00000000 34a2a6a5 00000001 00000000 00000000 |4...............| 00000010 00000000 00000000 0000102f |.........../ | -------- \---This word is the port the caller should use to connect/talk to the requested service. What you'll typically see next is a new session started to this port. x102f=4143. ---------------------------------------------------------------------