Logical Volume Striping

AIX Versions 3.2 and 4 Performance Tuning Guide

Logical Volume Striping

Striping is a technique for spreading the data in a logical volume across several disk drives in such a way that the I/O capacity of the disk drives can be used in parallel to access data on the logical volume. (The ability to create striped logical volumes is not available on AIX Version 3.2.5.) The primary objective of striping is very high-performance reading and writing of large sequential files. The figure "Striped Logical Volume" /dev/lvs0 gives a simple example.

In an ordinary logical volume, the data addresses correspond to the sequence of blocks in the underlying physical partitions. In a striped logical volume, the data addresses follow the sequence of stripe units. A complete stripe consists of one stripe unit on each of the physical devices that contains part of the striped logical volume. The LVM determines which physical blocks on which physical drives correspond to a block being read or written. If more than one drive is involved, the necessary I/O operations are scheduled for all drives simultaneously.

As an example, suppose that the hypothetical lvs0 has a stripe-unit size of 64KB, consists of six 2MB partitions, and contains a journaled file system (JFS). If an application is reading a large sequential file and read-ahead has reached a steady state, each read will result in two or three I/Os being scheduled to each of the disk drives to read a total of eight pages (assuming that the file is on consecutive blocks in the logical volume). The read operations are performed in the order determined by the disk device driver. The requested data is assembled from the various pieces of input and returned to the application.

Although each disk device will have a different initial latency, depending on where its accessor was at the beginning of the operation, once the process reaches a steady state, all three disks should be reading at close to their maximum speed.

Designing a Striped Logical Volume

When a striped logical volume is defined, you specify:

drives	Obviously, at least two physical drives are required. The drives used should have little other activity when the performance-critical sequential I/O is taking place. Some combinations of disk adapter and disk drive will require dividing the workload of a striped logical volume between two or more adapters.
stripe unit size	Although this can be any power of 2 from 4KB through 128KB, you should take sequential read ahead into account, since that will be the mechanism that issues most of the reads. The objective is to have each read-ahead operation result in at least one I/O, ideally an equal number, to each disk drive.
size	The number of physical partitions allocated to the logical volume must be an integral multiple of the number of disk drives used.
attributes	Cannot be mirrored; that is, copies = 1.

Tuning for Striped Logical Volume I/O

In benchmarking situations, the following techniques have yielded the highest levels of sequential I/O throughput:

Stripe unit size of 64KB.
max_coalesce of 64KB (the default). Equal to the stripe unit size.
minpgahead of 2.
maxpgahead of (16 times the number of disk drives). This causes page ahead to be done in units of the stripe unit size (64KB) times the number of disk drives, resulting in the reading of one stripe unit from each disk drive for each read ahead operation.
I/O requests for (64KB times the number of disk drives). This is equal to the maxpgahead value.
Modify maxfree to accommodate the change in maxpgahead. See "Choosing minfree and maxfree Settings".
64-byte aligned I/O buffers. If the logical volume will occupy physical drives that are connected to two or more disk adapters, the I/O buffers used should be allocated on 64-byte boundaries. This avoids having the LVM serialize the I/Os to the different disks. The following code would yield a 64-byte-aligned buffer pointer:
```
char *buffer;
buffer = malloc(MAXBLKSIZE+64);
buffer = ((int)buffer + 64) & ~0x3f;
```

If the striped logical volumes are on raw logical volumes and writes larger than 1.125 MB are being done to these striped raw logical volumes, increasing the lvm_bufcnt parameter of vmtune might increase throughput of the write activity.