Chapter 10. DFS Performance Tuning

Note: The following recommendations are based on performance experiments using AIX Version 3.2.5. At the time this book was written, the degree to which these recommendations would apply to AIX Version 4 was not known.

From the performance standpoint, the most important difference between DCE Distributed File Service (DFS) and NFS is the client data-caching capability of DFS, so it is not surprising that the most important performance-tuning techniques for DFS involve choosing the attributes of the client cache. The client and server parameter choices discussed in this section are:

• "DFS Caching on Disk or Memory?"
• "DFS Cache Size"
• "DFS Cache Chunk Size"
• "Number of DFS Cache Chunks"
• "Location of DFS Disk Cache"
• "Cache Status Buffer Size"
• "Effect of Application Read/Write Size"
• "Communications Parameter Settings for DFS"
• "DFS File Server Tuning"
• "DCE LFS Tuning for DFS Performance"

DFS Caching on Disk or Memory?

To assess the disk versus memory trade-off in your environment, consider the following points:

• If the system being tuned, or another system with similar workload, is already running DFS with a disk cache, you can estimate the required size of a memory cache by issuing the following command toward the end of a period of peak workload:

  cm getcachesize

  Divide the number of 1KB blocks being used by .9 to determine the memory cache size needed to accommodate the same amount of data. (About 10% of the blocks in the cache are used for DFS record keeping.)
• If the data being handled is frequently reaccessed, the greater potential capacity of a disk cache is probably appropriate.
• If the data being handled is so extensive that it would overflow the largest feasible disk cache, or if the data is frequently changed by another client, a memory cache is probably more appropriate because of its greater effect on RPC performance.
• The size of a memory cache should not exceed 10% of the real memory size of the machine. The recommended size is about 5% of real memory. Because DFS exploits the memory caching capability of the AIX VMM, most of a DFS memory cache is used to hold directory and mount-point information.
• If your system shows any sign of being memory-bound, as evidenced by nonzero values in the pi or po columns of a vmstat report, you should not use a memory cache for DFS.
• As a feasibility check, you could temporarily reduce, using rmss, the effective memory size of the machine by the amount of memory you are considering using for a memory cache. If you observe paging activity, or diminished performance, or both, you should not use a memory cache. See "Assessing Memory Requirements via the rmss Command".

DFS Cache Size

Determining the appropriate DFS cache size for a particular system will take some experimentation. You might begin by estimating the sum of:

• The sizes of the set of DFS-resident data files that are read at least once a day.
• The amount of DFS-resident data that is generated by the users of the system each day.
• The sizes of the DFS-resident programs that are executed more than once a day.

If the users' home directories are in DFS, you will want to make an allowance for the frequency with which the home directory is accessed, and the effect on perceived responsiveness of the system.

The size of the client cache is specified in the CacheInfo file and can be overridden with the dfsd -blocks n option, where n is the number of KB in the cache. This parameter applies to both memory and disk caches.

DFS Cache Chunk Size

The DFS cache chunk size can range from 8KB to 256KB. For large files (several MB), sequential read and write performance increases as chunk size increases, up to about 64KB. For very large files (100MB or more) a chunk size of 256KB yields the best read performance.

The chunk size is specified with the dfsd -chunksize n option, where n is an integer from 13 to 18, inclusive. The cache size is 2**n bytes, and so ranges from 8KB (2**13) to 256KB(2**18). This parameter applies to both memory and disk caches. The default size is 8KB for memory caches and 64KB for disk caches.

Number of DFS Cache Chunks

This parameter only applies to disk caches. For memory caches, the number of chunks is already specified by the combination of cache size and chunk size. For disk caches, the default number of chunks is computed as the number of cache blocks divided by 8. If a du of the cache directory indicates that the space is less than 90% full, increase the number of cache chunks with the dfsd -files n option, where n is the number of chunks to be accommodated. This allows better utilization of the available cache space in applications that use many small files. Since multiple files cannot share a chunk, the number of chunks determines the maximum number of files the cache can accommodate.

Location of DFS Disk Cache

The disk cache should be in a logical volume that is:

• In the outer_edge area, if it is on a 200MB, 540MB, or 1.0GB disk drive.
• In the center area, if it is on any other disk drive.
• Not in the rootvg volume group.
• Not on the same physical volume as a paging space.
• Primarily or exclusively for use by the disk cache.
• Large enough to hold the specified disk cache without encroachment by other contents.

Cache Status-Buffer Size

The status-buffer size limits the maximum number of files that can be in the cache at one time. One entry is required for each file. If the status buffer is full, new files will displace old files in the cache, even though there is enough disk space to hold them. If your workload consists mostly of files that are equal to or smaller than the chunk size, the status buffer should have as many entries as there are chunks in the cache.

The status-buffer size is specified with the dfsd -stat n option, where n is the number of entries in the status buffer. The default value of n is 300.

Effect of Application Read/Write Size

Sequential read and write performance are affected by the size of the records being read or written by the application. In general, read throughput increases with record size up to 4KB, above which it levels off. Write throughput increases with record size up to 2KB, above which it levels off or decreases slightly.

Communications Parameter Settings for DFS

DFS uses UDP as its communications protocol. The recommendations for tuning DFS communications for servers and multiuser client systems parallel those for tuning communications in general (see "UDP, TCP/IP, and mbuf Tuning Parameters Summary"):

• Set the network adapter transmit and receive queue sizes to 150 ( the maximum). This can be done with smit commodev -> (adapter type) -> Adapter -> Change / Show Characteristics of a (adapter type) Adapter. These parameters cannot be changed while the adapter is in operation. SMIT allows you to specify the change to take effect when the system is next restarted.

  You can also use chdev to set these parameters, if you take the adapter offline first. For example, for a Token-Ring adapter, the sequence of commands would be:

  # ifconfig tr0 detach
  # chdev -l tok0 -a xmt_que_size=150 -a rec_que_size=150
  # ifconfig tr0 hostname up

  You can observe the effect of the change with:

  $ lsattr -E -l tok0

• If a netstat -s command reports a nonzero number in udp: n socket buffer overflows , increasing the sb_max and udp_recvspace parameters with the no command will only solve the problem if an application other than DFS is experiencing the overflows. DFS sets its own values (176KB) for sb_max and udp_recvspace. These values are not displayed or changed by the no command.

DFS File Server Tuning

On high-speed servers, it may be desirable to increase the number of -mainprocs and -tokenprocs (in the fxd command), to ensure that all of the available CPU capacity can be used effectively.

• A good level to start with is -mainprocs 10 -tokenprocs 4.
• Run vmstat during periods of heavy load. If a considerable level of CPU I/O wait is being experienced, try increasing the -mainprocs and -tokenprocs values further.

DCE LFS Tuning for DFS Performance

The following should be considered when setting up a DCE LFS aggregate (using the newaggr command) on a DFS server:

• If most of the files will be large, set the -blocksize parameter to the largest permitted value that is less than the typical file size. The -blocksize parameter can any power of 2 in the range from 4KB to 64KB.
• If most of the files will be several times larger than the -blocksize parameter, set the -fragsize parameter equal to the -blocksize parameter. This may use some additional disk space, but will streamline processing.
• If the aggregate is smaller than 100MB, use the -logsize parameter to ensure that the log is larger than the default (1% of the aggregate size). In general, logsize should never be less than 1000 blocks.