NFS gathers statistics on types of NFS operations performed, along with error information and performance indicators. You can use the following commands to identify network problems and observe the type of NFS operations taking place on your system.
The nfsstat command displays statistical information about the NFS and the RPC interface to the kernel for clients and servers. This command could also be used to reinitialize the counters for these statistics (nfsstat -z). For performance issues, the RPC statistics (-r option) are the first place to look. The NFS statistics show you how the applications use NFS.
The nfsstat command displays statistical information about RPC calls, such as:
The NFS part of the nfsstat command output is divided into Version 2 and Version 3 statistics of NFS. The RPC part is divided into Connection oriented (TCP) and Connectionless (UDP) statistics.
The NFS server displays the number of NFS calls received (calls) and rejected (badcalls) due to authentication, as well as the counts and percentages for the various kinds of calls made.
The following example shows the server part of the nfsstat command output specified by the -s option, as follows:
# nfsstat -s Server rpc: Connection oriented: calls badcalls nullrecv badlen xdrcall dupchecks dupreqs 15835 0 0 0 0 772 0 Connectionless: calls badcalls nullrecv badlen xdrcall dupchecks dupreqs 0 0 0 0 0 0 0 Server nfs: calls badcalls public_v2 public_v3 15835 0 0 0 Version 2: (0 calls) null getattr setattr root lookup readlink read 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% wrcache write create remove rename link symlink 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% mkdir rmdir readdir statfs 0 0% 0 0% 0 0% 0 0% Version 3: (15835 calls) null getattr setattr lookup access readlink read 7 0% 3033 19% 55 0% 1008 6% 1542 9% 20 0% 9000 56% write create mkdir symlink mknod remove rmdir 175 1% 185 1% 0 0% 0 0% 0 0% 120 0% 0 0% rename link readdir readdir+ fsstat fsinfo pathconf 87 0% 0 0% 1 0% 150 0% 348 2% 7 0% 0 0% commit 97 0%
RPC output for server (-s) is as follows:
The output also displays a count of the various kinds of calls and their respective percentages.
Duplicate checks are performed for operations that cannot be performed twice with the same result. The classic example is the rm command. The first rm command will succeed, but if the reply is lost, the client will retransmit it. We want duplicate requests like these to succeed, so the duplicate cache is consulted, and if it is a duplicate request, the same (successful) result is returned on the duplicate request as was generated on the initial request.
By looking at the percentage of calls for different types of operations (such as getattr(), read(), write(), or readdir()), you can decide what type of tuning to use. For example, if the percentage of getattr() calls is very high, then tuning attribute caches may be advantageous. If the percentage of write() calls is very high, then disk and LVM tuning is important. If the percentage of read() calls is very high, then using more RAM for caching files could improve performance.
The NFS client displays the number of calls sent and rejected, as well as the number of times a client handle was received (clgets) and a count of the various kinds of calls and their respective percentages.
The following example shows the nfsstat output specified for clients using the -c option, as follows:
# nfsstat -c Client rpc: Connection oriented calls badcalls badxids timeouts newcreds badverfs timers 0 0 0 0 0 0 0 nomem cantconn interrupts 0 0 0 Connectionless calls badcalls retrans badxids timeouts newcreds badverfs 6553 0 0 0 0 0 0 timers nomem cantsend 0 0 0 Client nfs: calls badcalls clgets cltoomany 6541 0 0 0 Version 2: (6541 calls) null getattr setattr root lookup readlink read 0 0% 590 9% 414 6% 0 0% 2308 35% 0 0% 0 0% wrcache write create remove rename link symlink 0 0% 2482 37% 276 4% 277 4% 147 2% 0 0% 0 0% mkdir rmdir readdir statfs 6 0% 6 0% 30 0% 5 0% Version 3: (0 calls) null getattr setattr lookup access readlink read 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% write create mkdir symlink mknod remove rmdir 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% rename link readdir readdir+ fsstat fsinfo pathconf 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% commit 0 0%
RPC output for the client (-c) is as follows:
The output also displays a count of the various kinds of calls and their respective percentages.
For performance monitoring, the nfsstat -c command provides information on whether the network is dropping UDP packets. A network may drop a packet if it cannot handle it. Dropped packets can be the result of the response time of the network hardware or software or an overloaded CPU on the server. Dropped packets are not actually lost, because a replacement request is issued for them.
The retrans column in the RPC section displays the number of times requests were retransmitted due to a timeout in waiting for a response. This situation is related to dropped UDP packets. If the retrans number consistently exceeds five percent of the total calls in column one, it indicates a problem with the server keeping up with demand. Use the vmstat, netpmon, and iostat commands on the server machine to check the load.
A high badxid count implies that requests are reaching the various NFS servers, but the servers are too loaded to send replies before the client's RPC calls time out and are retransmitted. The badxid value is incremented each time a duplicate reply is received for a transmitted request (an RPC request retains its XID through all transmission cycles). Excessive retransmissions place an additional strain on the server, further degrading response time. If badxid and timeouts are greater than five percent of the total calls, increase the timeo parameter of the NFS-mount options by using the smitty chnfsmnt command. If badxid is 0, but retrans and timeouts are sizable, attempt to decrease the NFS buffer size (that is, the rsize and wsize options of the mount command).
If the server is CPU-bound, NFS and its daemons are affected. To improve the situation, the server must be tuned or upgraded, or the user can localize the application files. If the server is I/O-bound, the server file systems can be reorganized, or localized files can be used.
If the number of retransmits and timeouts are close to the same value, it is certain that packets are being dropped. Packets are rarely dropped on the client. Usually, packets are dropped on either the network or on the server. The server could drop packets if it overflows its interface driver's transmit queue or if the server's User Datagram Protocol (UDP) socket buffer was overflown (nfs_socketsize). If there are no socket buffer overflows or Oerrs on the server, and the client is getting many retransmits and timeouts, packets are possibly being dropped on the network. Problems could occur in media and network devices, such as routers, bridges, or concentrators. Network sniffers and other tools can be used to debug such problems. See Dropped Packets for further discussion.
In some instances, an application or user experiences poor performance, yet examination of the nfsstat -c output indicates no or very few timeouts and retransmits. This means that the client is receiving responses from the server as fast as it is asking for them. The first thing to check is that there is an appropriate number of biod daemons running on the client machine. This can also be observed when an application is doing remote file locking. When remote file locks are set on a file served over NFS, the client goes into a fully synchronous mode of operation that will turn off all data and attribute caching for the file. The result is very slow performance and is, unfortunately, normal. Locking packets can be identified in ipreport output by looking for NLM requests.
The nfsstat -m command displays the server name and address, mount flags, current read and write sizes, retransmission count, and the timers used for dynamic retransmission for each NFS mount on the client, as follows:
# nfsstat -m /SAVE from /SAVE:itsorus.austin.ibm.com Flags: vers=2,proto=udp,auth=unix,soft,intr,dynamic,rsize=8192,wsize=8192,retrans=5 Lookups: srtt=27 (67ms), dev=17 (85ms), cur=11 (220ms) Reads: srtt=16 (40ms), dev=7 (35ms), cur=5 (100ms) Writes: srtt=42 (105ms), dev=14 (70ms), cur=12 (240ms) All: srtt=27 (67ms), dev=17 (85ms), cur=11 (220ms)
The numbers in parentheses in the example output are the actual times in milliseconds. The other values are unscaled values kept by the operating system kernel. You can ignore the unscaled values. Response times are shown for lookups, reads, writes, and a combination of all of these operations (All). Other definitions used in this output are as follows:
See The netpmon Command for a discussion of this command and its output.
The nfso command can be used to configure NFS attributes. It sets or displays network options in the currently running kernel. Therefore, the command must run after each system startup or network configuration.
The nfso parameters and their values can be displayed by using the nfso -a command, as follows:
(tremor:) # nfso -a portcheck= 0 udpchecksum= 1 nfs_socketsize= 60000 nfs_tcp_socketsize= 60000 nfs_setattr_error= 0 nfs_gather_threshold= 4096 nfs_repeat_messages= 0 nfs_udp_duplicate_cache_size= 5000 nfs_tcp_duplicate_cache_size= 5000 nfs_server_base_priority= 0 nfs_dynamic_retrans= 1 nfs_iopace_pages= 0 nfs_max_connections= 1024 nfs_max_threads= 128 nfs_use_reserved_ports= 0 nfs_device_specific_bufs= 1 nfs_server_clread= 1 nfs_rfc1323= 1 nfs_max_write_size= 0 nfs_max_read_size= 0 nfs_allow_all_signals= 0 nfs_v2_pdts= 1 nfs_v3_pdts= 1 nfs_v2_vm_bufs= 1000 nfs_v3_vm_bufs= 1000 nfs_securenfs_authtimeout= 0 nfs_v3_server_readdirplus= 1
For a description of these attributes, see Network Tunable Parameters. Most NFS attributes are run-time attributes that can be changed at any time. Load time attributes, such as nfs_socketsize, need NFS to be stopped first and restarted afterwards.
To display or change a specific parameter, use the nfso -o command, as follows:
# nfso -o portcheck portcheck= 0 # nfso -o portcheck=1
The parameters can be reset to their default value by using the -d option, as follows:
# nfso -d portcheck # nfso -o portcheck portcheck= 0
Following is a summary of NFS-related files, commands, daemons, and subroutines. See the AIX 5L Version 5.2 System Management Guide: Communications and Networks and the AIX 5L Version 5.2 Commands Reference for details.
Following is a list of NFS files containing configuration information:
Following is a list of NFS commands:
Following is a list of NFS locking daemons:
Following is a list of network service daemons and utilities:
Following is a list of secure networking daemons and utilities:
Following is a diskless client support configuration file:
Following is a list of NFS subroutines: