filemon Command

AIX Version 4.3 Commands Reference, Volume 2

filemon Command

Purpose

Monitors the performance of the file system, and reports the I/O activity on behalf of logical files, virtual memory segments, logical volumes, and physical volumes.

Syntax

filemon [ -d ] [ -i File ] [ -o File ] [ -O Levels ] [ -P ] [ -T n ] [ -u ] [ -v ]

Description

The filemon command monitors a trace of file system and I/O system events, and reports on the file and I/O access performance during that period.

In its normal mode, the filemon command runs in the background while one or more application programs or system commands are being executed and monitored. The filemon command automatically starts and monitors a trace of the program's file system and I/O events in real time. By default, the trace is started immediately; optionally, tracing may be deferred until the user issues a trcon command. The user can issue trcoff and trcon commands while the filemon command is running in order to turn off and on monitoring, as desired. When tracing is stopped by a trcstop command, the filemon command generates an I/O activity report and exits.

The filemon command can also process a trace file that has been previously recorded by the AIX trace facility. The file and I/O activity report will be based on the events recorded in that file.

To provide a more complete understanding of file system performance for an application, the filemon command monitors file and I/O activity at four levels:

Logical file system: The filemon command monitors logical I/O operations on logical files. The monitored operations include all read , write , open , and lseek system calls, which may or may not result in actual physical I/O, depending on whether or not the files are already buffered in memory. I/O statistics are kept on a per-file basis.
Virtual memory system: The filemon command monitors physical I/O operations (that is, paging) between segments and their images on disk. I/O statistics are kept on a per-segment basis.
Logical volumes: The filemon command monitors I/O operations on logical volumes. I/O statistics are kept on a per-logical-volume basis.
Physical volumes: The filemon command monitors I/O operations on physical volumes. At this level, physical resource utilizations are obtained. I/O statistics are kept on a per-physical-volume basis.

Any combination of the four levels can be monitored, as specified by the command line flags. By default, the filemon command only monitors I/O operations at the virtual memory, logical volume, and physical volume levels. These levels are all concerned with requests for real disk I/O.

The filemon command writes its report to standard output or to a specified file. The report begins with a summary of the I/O activity for each of the levels being monitored and ends with detailed I/O activity statistics for each of the levels being monitored. Summary and detailed report contents are described in the "Reports " section.

Notes:The reports produced by the filemon command can be quite long. Consequently, the -o option should usually be used to write the report to an output file. When a physical device is opened and accessed directly by an application, only reads and writes of complete 512-byte blocks are reflected in the report. "Short" reads and writes, used by the device driver to issue device commands and read device status, are ignored. CD-ROMs do not have concentric "tracks" or "cylinders," as in hard files. (There is one spiral track.) Consequently, it is not possible to report seek distance statistics for CD-ROMs in terms of cylinders. The -u flag is used to generate reports on files opened prior to the start of the trace daemon. Some of this data can be useful, but much of it applies to daemons and other unrelated activity. This background information can be overwhelming, especially on large systems. If the /unix file and the running kernel are not the same, then the kernel addresses will be incorrect, causing the filemon command to exit. When using the filemon command from within a shell script, allow for a slight delay prior to viewing the contents of the filemon output file. The filemon command may take a few seconds to produce this report.

System Trace Facility

The filemon command obtains raw I/O performance data using the AIX system trace facility. Currently, the trace facility only supports one output stream. Consequently, only one filemon or trace process can be active at a time. If another filemon or trace process is already running, the filemon command will respond with the message:

/dev/systrace: Device busy

While monitoring very I/O-intensive applications, the filemon command may not be able to consume trace events as fast as they are produced in real time. When that happens, the error message:

Trace kernel buffers overflowed, N missed entries

will be displayed on stderr , indicating how many trace events were lost while the trace buffers were full. The filemon command will continue monitoring I/O activity, but the accuracy of the report will be diminished to some unknown degree. One way to prevent overflow is to monitor fewer levels of the file and I/O subsystems: the number of trace events generated is proportional to the number of levels monitored. Additionally, the trace buffer size can be increased using the -T option, to accommodate larger bursts of trace events before overflow. Remember that increasing the trace buffer size will result in more pinned memory, and therefore may effect I/O and paging behavior.

In memory-constrained environments (where demand for memory exceeds supply), the -P option can be used to pin the text and data pages of the real-time filemon process in memory so the pages cannot be swapped out. If the -P option is not used, allowing the filemon process to be swapped out, the progress of the filemon command may be delayed to the point where it cannot process trace events fast enough. This situation leads to trace buffer overflow as described above. Of course, pinning this process takes memory away from the application (although the filemon command is not a large program, its process image can consume up to 500KB).

Before using the filemon command to process an existing trace data file, you must use the -r option of the trcrpt command to rewrite the trace data sequentially to a new file. Otherwise, the filemon command produces the following error message, and then exits:

error: run 'trcrpt -r' on logfile first

Reports

Each report generated by the filemon command has a header that identifies the date, the machine ID, and the length of the monitoring period, in seconds. The CPU utilization during the monitoring period is also reported.

Next, summary reports are generated for each of the file system levels being monitored. By default, the logical file and virtual memory reports are limited to the 20 most active files and segments, respectively, as measured by the total amount of data transferred. If the -v flag has been specified, activity for all files and segments is reported. There is one row for each reported file, segment, or volume. The columns in each row for the four summary reports are described in the following lists:

Most Active Files Report

Column: Description
#MBS: Total number of megabytes transferred to/from file. The rows are sorted by this field, in decreasing order.
#opns: Number of times the file was opened during measurement period.
#rds: Number of read system calls made against file.
#wrs: Number of write system calls made against file.
file: Name of file (full path name is in detailed report).
volume:inode: Name of volume which contains the file, and the file's i-node number. This field can be used to associate a file with its corresponding persistent segment, shown in the virtual memory I/O reports. This field may be blank; for example, for temporary files created and deleted during execution.

Most Active Segments Report

Column

Description

#MBS

Total number of megabytes transferred to/from segment. The rows are sorted by this field, in decreasing order.

#rpgs

Number of 4096-byte pages read into segment from disk (that is page).

#wpgs

Number of 4096-byte pages written from segment to disk (page out).

segid

Internal ID of segment.

segtype

Type of segment: working segment, persistent segment (local file), client segment (remote file), page table segment, system segment, or special persistent segments containing file system data (log, root directory, .inode , .inodemap , .inodex , .inodexmap , .indirect , .diskmap ).

volume:inode

For persistent segments, name of volume that contains the associated file, and the file's inode number. This field can be used to associate a persistent segment with its corresponding file, shown in the file I/O reports. This field will be blank for non-persistent segments.

Note: The virtual memory analysis tool, svmon can be used to display more information about a segment, given its segment ID (segid), as follows:

svmon -S <segid>

Most Active Logical Volumes Report

Column: Description
util: Utilization of the volume (fraction of time busy). The rows are sorted by this field, in decreasing order.
#rblk: Number of 512-byte blocks read from the volume.
#wblk: Number of 512-byte blocks written to the volume.
KB/sec: Total transfer throughput, in Kilobytes per second.
volume: Name of volume.
description: Contents of volume: either a file system name, or logical volume type (paging, jfslog, boot, or sysdump). Also, indicates if the file system is fragmented or compressed.

Most Active Physical Volumes Report

Column: Description
uti l: Utilization of the volume (fraction of time busy). The rows are sorted by this field, in decreasing order.
#rblk: Number of 512-byte blocks read from the volume.
#wblk: Number of 512-byte blocks written to the volume.
KB/sec: Total volume throughput, in Kilobytes per second.
volume: Name of volume.
description: Type of volume, for example, 120MB disk , 355MB SCSI , or CDROM SCSI .
Note: Logical volume I/O requests start before, and end after, physical volume I/O requests. For that reason, total logical volume utilization will appear to be higher than total physical volume utilization.

Finally, detailed reports are generated for each of the file system levels being monitored. By default, the logical file and virtual memory reports are limited to the 20 most active files and segments, respectively, as measured by the total amount of data transferred. If the -v flag has been specified, activity for all files and segments is reported. There is one entry for each reported file, segment, or volume. The fields in each entry are described below for the four detailed reports are described in the following lists.

Some of the fields report a single value, others report statistics that characterize a distribution of many values. For example, response time statistics are kept for all read or write requests that were monitored. The average, minimum, and maximum response times are reported, as well as the standard deviation of the response times. The standard deviation is used to show how much the individual response times deviated from the average. Roughly two-thirds of the sampled response times are between average - standard deviation and average + standard deviation . If the distribution of response times is scattered over a large range, the standard deviation will be large compared to the average response time.

Detailed File Stats Report

Column: Description
FILE: Name of the file. The full path name is given, if possible.
volume: Name of the logical volume/file system containing the file.
inode: I-node number for the file within its file system.
opens: Number of times the file was opened while monitored.
total bytes xfrd: Total number of bytes read/written to/from the file.
reads: Number of read calls against the file.
read sizes (bytes): The read transfer-size statistics (avg/min/max/sdev), in bytes.
read times (msec): The read response-time statistics (avg/min/max/sdev), in milliseconds.
writes: Number of write calls against the file.
write sizes (bytes): The write transfer-size statistics.
write times (msec): The write response-time statistics.
seeks: Number of lseek subroutine calls.

Detailed VM Segment Stats Report

Column: Description
SEGMENT: Internal AIX segment ID.
segtype: Type of segment contents.
segment flags: Various segment attributes.
volume: For persistent segments, the name of the logical volume containing the corresponding file.
inode: For persistent segments, the i-node number for the corresponding file.
reads: Number of 4096-byte pages read into the segment (that is, paged in).
read times (msec): The read response-time statistics (avg/min/max/sdev), in milliseconds.
read sequences: Number of read sequences. A sequence is a string of pages that are read (paged in) consecutively. The number of read sequences is an indicator of the amount of sequential access.
read seq. lengths: Statistics describing the lengths of the read sequences, in pages.
writes: Number of pages written from the segment (that is, paged out).
write times (msec): Write response time statistics.
write sequences: Number of write sequences. A sequence is a string of pages that are written (paged out) consecutively.
write seq. lengths: Statistics describing the lengths of the write sequences, in pages.

Detailed Logical/Physical Volume Stats Reports

Column: Description
VOLUME: Name of the volume.
description: Description of the volume. (Describes contents, if discussing a logical volume; describes type, if dealing with a physical volume.)
reads: Number of read requests made against the volume.
read sizes (blks): The read transfer-size statistics (avg/min/max/sdev), in units of 512-byte blocks.
read times (msec): The read response-time statistics (avg/min/max/sdev), in milliseconds.
read sequences: Number of read sequences. A sequence is a string of 512-byte blocks that are read consecutively and indicate the amount of sequential access.
read seq. lengths: Statistics describing the lengths of the read sequences, in blocks.
writes: Number of write requests made against the volume.
write sizes (blks): The write transfer-size statistics.
write times (msec): The write-response time statistics.
write sequences: Number of write sequences. A sequence is a string of 512-byte blocks that are written consecutively.
write seq. lengths: Statistics describing the lengths of the write sequences, in blocks.
seeks: Number of seeks that preceded a read or write request; also expressed as a percentage of the total reads and writes that required seeks.
seek dist (blks): Seek distance statistics, in units of 512-byte blocks. In addition to the usual statistics (avg/min/max/sdev), the distance of the initial seek operation (assuming block 0 was the starting position) is reported separately. This seek distance is sometimes very large, so it is reported separately to avoid skewing the other statistics.
seek dist (cyls): (Hard files only.) Seek distance statistics, in units of disk cylinders.
time to next req: Statistics (avg/min/max/sdev) describing the length of time, in milliseconds, between consecutive read or write requests to the volume. This column indicates the rate at which the volume is being accessed.
throughput: Total volume throughput, in Kilobytes per second.
utilization: Fraction of time the volume was busy. The entries in this report are sorted by this field, in decreasing order.

Flags

-i File

Reads the I/O trace data from the specified File, instead of from the real-time trace process. The filemon report summarizes the I/O activity for the system and period represented by the trace file.

Note: Trace data files are usually written in a circular manner. If the trace data has wrapped around, the chronological beginning and end of the trace may occur in the middle of the file. Use the raw mode of the trcrpt command to rewrite the data sequentially, before invoking the filemon command, as follows:

trcrpt -r file >
 new.file

For the report to be accurate, the trace file must contain all the hooks required by the filemon command.

-o File

Writes the I/O activity report to the specified File, instead of to the stdout file.

-d

Starts the filemon command, but defers tracing until the trcon command has been executed by the user. By default, tracing is started immediately.

-T n

Sets the kernel's trace buffer size to n bytes. The default size is 32,000 bytes. The buffer size can be increased to accommodate larger bursts of events, if any. (A typical event record size is 30 bytes.)

Note: The trace driver in the kernel uses double buffering, so in fact there will be two buffers allocated of size n bytes. Also, note that these buffers are pinned in memory, so they are not subject to paging. Large buffers may affect the performance of paging and other I/O.

-P

Pins monitor process in memory. The -P flag causes the filemon command's text and data pages to be pinned in memory for the duration of the monitoring period. This flag can be used to ensure that the real-time filemon process is not paged out when running in a memory-constrained environment.

-v

Prints extra information in the report. The most significant effect of the -v flag is that all logical files and all segments that were accessed are included in the I/O activity report, instead of only the 20 most active files and segments.

-O Levels

Monitors only the specified file system levels. Valid level identifiers are:

lf

Logical file level

vm

Virtual memory level

lv

Logical volume level

pv

Physical volume level

all: Short for lf,vm,lv,pv

The vm, lv, and pv levels are implied by default.

-u

Reports on files that were opened prior to the start of the trace daemon. The process ID (PID) and the file descriptor (FD) are substituted for the file name.

Note: Since PIDs and FDs are reusable, it is possible to see different files reported with the same name field.

Examples

To monitor the physical I/O activity of the virtual memory, logical volume, and physical volume levels of the file system, enter:
```
filemon
```
The filemon command automatically starts the system trace and puts itself in the background. After this command, enter the application programs and system commands to be run at this time, then enter:
```
trcstop
```
After the trcstop command is issued, the I/O activity report is displayed on standard output (but will probably scroll off the screen). The virtual memory I/O report will be limited to the 20 segments that incurred the most I/O.
To monitor the activity at all file system levels, and write the report to the fmon.out file, enter:
```
filemon -o fmon.out -O all
```
The filemon command automatically starts the system trace and puts itself in the background. After this command, enter the application programs and system commands to be run at this time, then enter:
```
trcstop
```
After the trcstop command is issued, the I/O activity report is written to the fmon.out file . All four levels of the file and I/O system (the logical file, virtual memory, logical volume, and physical volume levels) will be monitored. The logical file and virtual memory I/O reports will be limited to the 20 files and segments (respectively) that incurred the most I/O.
To monitor the activity at all file system levels and write a verbose report to the fmon.out file, enter:
```
filemon -v -o fmon.out -O all
```
The filemon command automatically starts the system trace and puts itself in the background. After this command, enter the application programs and system commands to be run at this time, then enter:
```
trcstop
```
This example is similar to the previous example, except a verbose report is generated on the fmon.out file. The primary difference is that the filemon command will indicate the steps it is taking to start up the trace, and the summary and detailed reports will include all files and segments that incurred any I/O (there may be many), instead of just the top 20.
To report on I/O activity captured by a previously recorded trace session, enter:
```
filemon -i trcfile | pg
```
In this example, the filemon command reads file system trace events from the input file trcfile . The input file must already be in raw trace format, as a result of running the trcrpt -r command. Since the trace data is already captured on a file, the filemon command does not put itself in the background to allow application programs to be run. After the entire file is read, an I/O activity report for the virtual memory, logical volume, and physical volume levels will be displayed on standard output (which, in this example, is piped to pg ).
To monitor the I/O activity for logical and physical volumes only, while controlling the monitored intervals using the trcon and trcoff commands, enter:
```
filemon -d -o fmon.out -O pv,lv
```
The filemon command automatically starts the system trace and puts itself in the background. After this command,you can enter the unmonitored application programs and system commands to be run at this time, then enter:
```
trcon
```
After this command,you can enter the monitored application programs and system commands to be run at this time, then enter:
```
trcoff
```
After this command,you can enter the unmonitored application programs and system commands to be run at this time, then enter:
```
trcon
```
After this command,you can enter the monitored application programs and system commands to be run at this time, then enter:
```
trcstop
```
In this example, the -O flag is used to restrict monitoring to logical and physical volumes only. Only those trace events that are relevant to logical and physical volumes are enabled. Also, as a result of using the -d flag, monitoring is initially deferred until the trcon command is issued. System tracing can be intermittently disabled and reenabled using the trcoff and trcon commands, so that only specific intervals are monitored.

Related Information

The svmon command, trcrpt command, trcstop command.

The lseek subroutine.

Monitoring and Tuning Disk I/O in AIX Versions 3.2 and 4 Performance Tuning Guide