Performance Management Guide

Using Disk-I/O Pacing

Disk-I/O pacing is intended to prevent programs that generate very large amounts of output from saturating the system's I/O facilities and causing the response times of less-demanding programs to deteriorate. Disk-I/O pacing enforces per-segment (which effectively means per-file) high- and low-water marks on the sum of all pending I/Os. When a process tries to write to a file that already has high-water mark pending writes, the process is put to sleep until enough I/Os have completed to make the number of pending writes less than or equal to the low-water mark. The logic of I/O-request handling does not change. The output from high-volume processes is slowed down somewhat.

The high- and low-water marks are set with the SMIT command by selecting System Environments -> Change / Show Characteristics of Operating System(smitty chgsys) and then entering the number of pages for the high- and low-water marks. The default value for the high- and low-water marks is 0, which disables pacing. Changes to the maxpout and minpout values take effect immediately and remain in place until they are explicitly changed.

Example

The effect of pacing on performance can be demonstrated with an experiment that consists of starting a vi editor session on a new file while another process is copying a 64 MB file with the cp command. The file is copied from hdisk1 to hdisk0 and the vi executable program is located on hdisk0. For the vi session to start, it must page itself in as well as perform a few other I/Os, which it does randomly one page at a time. This takes about 50 physical I/Os, which can be completed in 0.71 seconds on a slow machine when there is no contention for the disk. With the high-water mark set to the default of 0, the logical writes from the cp command run ahead of the physical writes, and a large queue builds up.

Each I/O started by the vi session must wait its turn in the queue before the next I/O can be issued, and thus the vi command is not able to complete its needed I/O until after the cp command finishes. The following table shows the elapsed seconds for cp execution and vi initialization with different pacing parameters.

It is important to notice that the cp duration is always longer when pacing is set. Pacing sacrifices some throughput on I/O-intensive programs to improve the response time of other programs. The challenge for a system administrator is to choose settings that result in a throughput/response-time trade-off that is consistent with the organization's priorities.

The high- and low-water marks were chosen by trial and error, based on our knowledge of the I/O path. Choosing them is not straightforward because of the combination of write-behind and asynchronous writes. High-water marks of (4 * n) + 1, where n is a positive integer, work particularly well, because of the following interaction:

• The write-behind feature sends the previous four pages to disk when a logical write occurs to the first byte of the fifth page.
• If the pacing high-water mark were a multiple of 4 (for example, 8), a process would hit the high-water mark when it requested a write that extended into the 9th page. It would then be put to sleep, before the write-behind algorithm had an opportunity to detect that the fourth dirty page is complete and the four pages were ready to be written.
• The process would then sleep with four full pages of output until its outstanding writes fell below the pacing low-water mark.
• If, on the other hand, the high-water mark had been set to 9, write-behind would get to schedule the four pages for output before the process was suspended.

One limitation of pacing is that it does not offer as much control when a process writes buffers larger than 4 KB. When a write is sent to the VMM and the high-water mark has not been met, the VMM performs start I/Os on all pages in the buffer, even if that results in exceeding the high-water mark. Pacing works well on the cp command because the cp command writes 4 KB at a time; but if the cp command wrote larger buffers, the times shown in the previous table for starting the vi session would increase.

Disk-I/O pacing is a tuning parameter that can improve interactive response time in some situations where foreground or background programs that write large volumes of data are interfering with foreground requests. If not used properly, however, it can reduce throughput excessively. The settings in the previous table are a good place to start, but some experimenting will be needed to find the best settings for your workload. For the newer systems, view these numbers as the minimum starting point.

Programs whose presence in a workload may make imposition of disk-I/O pacing necessary include:

• Programs that generate large amounts of output algorithmically, and thus are not constrained by the time required to read input. Some such programs may need pacing on comparatively fast processors and not need it on comparatively slow processors.
• Programs that write large, possibly somewhat modified, files that have been read in their entirety shortly before writing begins (by a previous command, for example).
• Filters, such as the tar command, that read a file and write it out again with little processing. The need for pacing can be exacerbated if the input is being read from a faster disk drive than the output is being written to.