[ Bottom of Page | Previous Page | Next Page | Contents | Index | Library Home | Legal | Search ]

Commands Reference, Volume 2

filemon Command

Purpose

Monitors the performance of the file system, and reports the I/O activity on behalf of logical files, virtual memory segments, logical volumes, and physical volumes.

Syntax

filemon [ -d ] [ -i Trace_File -n Gennames_File] [ -o File] [ -O Levels] [ -P ] [ -T n] [ -u ] [ -v ]

Description

The filemon command monitors a trace of file system and I/O system events, and reports on the file and I/O access performance during that period.

In its normal mode, the filemon command runs in the background while one or more application programs or system commands are being executed and monitored. The filemon command automatically starts and monitors a trace of the program's file system and I/O events in real time. By default, the trace is started immediately; optionally, tracing may be deferred until the user issues a trcon command. The user can issue trcoff and trcon commands while the filemon command is running in order to turn off and on monitoring, as desired. When tracing is stopped by a trcstop command, the filemon command generates an I/O activity report and exits.

The filemon command can also process a trace file that has been previously recorded by the trace facility. The file and I/O activity report will be based on the events recorded in that file.

To provide a more complete understanding of file system performance for an application, the filemon command monitors file and I/O activity at four levels:

Logical file system The filemon command monitors logical I/O operations on logical files. The monitored operations include all read, write, open, and lseek system calls, which may or may not result in actual physical I/O, depending on whether or not the files are already buffered in memory. I/O statistics are kept on a per-file basis.
Virtual memory system The filemon command monitors physical I/O operations (that is, paging) between segments and their images on disk. I/O statistics are kept on a per-segment basis.
Logical volumes The filemon command monitors I/O operations on logical volumes. I/O statistics are kept on a per-logical-volume basis.
Physical volumes The filemon command monitors I/O operations on physical volumes. At this level, physical resource utilizations are obtained. I/O statistics are kept on a per-physical-volume basis.

Any combination of the four levels can be monitored, as specified by the command line flags. By default, the filemon command only monitors I/O operations at the virtual memory, logical volume, and physical volume levels. These levels are all concerned with requests for real disk I/O.

The filemon command writes its report to standard output or to a specified file. The report begins with a summary of the I/O activity for each of the levels being monitored and ends with detailed I/O activity statistics for each of the levels being monitored. Summary and detailed report contents are described in the Reports section.

Notes:
  1. The reports produced by the filemon command can be quite long. Consequently, the -o option should usually be used to write the report to an output file. When a physical device is opened and accessed directly by an application, only reads and writes of complete 512-byte blocks are reflected in the report. "Short" reads and writes, used by the device driver to issue device commands and read device status, are ignored. CD-ROMs do not have concentric "tracks" or "cylinders," as in hard files. (There is one spiral track.) Consequently, it is not possible to report seek distance statistics for CD-ROMs in terms of cylinders.
  2. The -u flag is used to generate reports on files opened prior to the start of the trace daemon. Some of this data can be useful, but much of it applies to daemons and other unrelated activity. This background information can be overwhelming, especially on large systems. If the /unix file and the running kernel are not the same, then the kernel addresses will be incorrect, causing the filemon command to exit. When using the filemon command from within a shell script, allow for a slight delay prior to viewing the contents of the filemon output file. The filemon command may take a few seconds to produce this report.

System Trace Facility

The filemon command obtains raw I/O performance data using the system trace facility. Currently, the trace facility only supports one output stream. Consequently, only one filemon or trace process can be active at a time. If another filemon or trace process is already running, the filemon command responds with the message:

/dev/systrace: Device busy

While monitoring very I/O-intensive applications, the filemon command may not be able to consume trace events as fast as they are produced in real time. When that happens, the error message:

Trace kernel buffers overflowed, N missed entries

will be displayed on stderr, indicating how many trace events were lost while the trace buffers were full. The filemon command will continue monitoring I/O activity, but the accuracy of the report will be diminished to some unknown degree. One way to prevent overflow is to monitor fewer levels of the file and I/O subsystems: the number of trace events generated is proportional to the number of levels monitored. Additionally, the trace buffer size can be increased using the -T option, to accommodate larger bursts of trace events before overflow. Remember that increasing the trace buffer size will result in more pinned memory, and therefore may effect I/O and paging behavior.

In memory-constrained environments (where demand for memory exceeds supply), the -P option can be used to pin the text and data pages of the real-time filemon process in memory so the pages cannot be swapped out. If the -P option is not used, allowing the filemon process to be swapped out, the progress of the filemon command may be delayed to the point where it cannot process trace events fast enough. This situation leads to trace buffer overflow as described above. Of course, pinning this process takes memory away from the application (although the filemon command is not a large program, its process image can consume up to 500KB).

Before using the filemon command to process an existing trace data file, you must use the -r option of the trcrpt command to rewrite the trace data sequentially to a new file. Otherwise, the filemon command produces the following error message, and then exits:

error: run 'trcrpt -r' on logfile first

The -i Trace_File and -n Gennames_File flags allow for offline processing by filemon of trace data files created by the trace command. Both flags must be supplied if either is present. These flags are useful when it is necessary to postprocess a trace file from a remote machine or perform the trace data collection at one time and postprocess it at another time. The flags are also useful when system load is high and trace hooks are being missed by filemon.

The gennames file (containing filesystem information) must be used from the machine that the trace came from. Also, it is wise to run gennames at close to the same time that the system trace file is created, so that the system configuration is the same for both.

Trace hooks relevant to filemon must be collected by the trace command and are specified by the trace -j flag. The relevant trace hooks are listed when filemon is invoked with the -v flag. The gennames command with -foption is then executed, with its output saved in Gennames_File to collect additional information for filemon. The -f option is used with the gennames command to collect the device information for physical and logical volumes. It is also used to get the virtual file system information used by offline filemon. Once the trace command has been executed, trcrpt -r must be run on the trace logfile and redirected to another file. Then this file and the Gennames_File may be provided to filemon.

Reports

Each report generated by the filemon command has a header that identifies the date, the machine ID, and the length of the monitoring period, in seconds. The CPU utilization during the monitoring period is also reported.

Next, summary reports are generated for each of the file system levels being monitored. By default, the logical file and virtual memory reports are limited to the 20 most active files and segments, respectively, as measured by the total amount of data transferred. If the -v flag has been specified, activity for all files and segments is reported. There is one row for each reported file, segment, or volume. The columns in each row for the four summary reports are described in the following lists:

Most Active Files Report

Column Description
#MBS Total number of megabytes transferred to/from file. The rows are sorted by this field, in decreasing order.
#opns Number of times the file was opened during measurement period.
#rds Number of read system calls made against file.
#wrs Number of write system calls made against file.
file Name of file (full path name is in detailed report).
volume:inode Name of volume that contains the file, and the file's i-node number. This field can be used to associate a file with its corresponding persistent segment, shown in the virtual memory I/O reports. This field may be blank; for example, for temporary files created and deleted during execution.

Most Active Segments Report

Column Description
#MBS Total number of megabytes transferred to/from segment. The rows are sorted by this field, in decreasing order.
#rpgs Number of 4096-byte pages read into segment from disk (that is, page).
#wpgs Number of 4096-byte pages written from segment to disk (page out).
segid Internal ID of segment.
segtype Type of segment: working segment, persistent segment (local file), client segment (remote file), page table segment, system segment, or special persistent segments containing file system data (log, root directory, .inode, .inodemap, .inodex, .inodexmap, .indirect, .diskmap).
volume:inode For persistent segments, name of volume that contains the associated file, and the file's inode number. This field can be used to associate a persistent segment with its corresponding file, shown in the file I/O reports. This field is blank for non-persistent segments.
Note
The virtual memory analysis tool, svmon can be used to display more information about a segment, given its segment ID (segid), as follows:
svmon -S <segid>

Most Active Logical Volumes Report

Column Description
util Utilization of the volume (fraction of time busy). The rows are sorted by this field, in decreasing order.
#rblk Number of 512-byte blocks read from the volume.
#wblk Number of 512-byte blocks written to the volume.
KB/sec Total transfer throughput, in Kilobytes per second.
volume Name of volume.
description Contents of volume: either a file system name, or logical volume type (paging, jfslog, boot, or sysdump). Also, indicates if the file system is fragmented or compressed.

Most Active Physical Volumes Report

Column Description
util Utilization of the volume (fraction of time busy). The rows are sorted by this field, in decreasing order.
#rblk Number of 512-byte blocks read from the volume.
#wblk Number of 512-byte blocks written to the volume.
KB/sec Total volume throughput, in Kilobytes per second.
volume Name of volume.
description Type of volume, for example, 120MB disk, 355MB SCSI, or CDROM SCSI.
Note
Logical volume I/O requests start before, and end after, physical volume I/O requests. For that reason, total logical volume utilization will appear to be higher than total physical volume utilization.

Finally, detailed reports are generated for each of the file system levels being monitored. By default, the logical file and virtual memory reports are limited to the 20 most active files and segments, respectively, as measured by the total amount of data transferred. If the -v flag is specified, activity for all files and segments is reported. There is one entry for each reported file, segment, or volume.

Some of the fields report a single value, others report statistics that characterize a distribution of many values. For example, response time statistics are kept for all read or write requests that were monitored. The average, minimum, and maximum response times are reported, as well as the standard deviation of the response times. The standard deviation is used to show how much the individual response times deviated from the average. Roughly two-thirds of the sampled response times are between average - standard deviation and average + standard deviation. If the distribution of response times is scattered over a large range, the standard deviation will be large compared to the average response time. The four detailed reports are described in the following lists:

Detailed File Stats Report

Column Description
FILE Name of the file. The full path name is given, if possible.
volume Name of the logical volume/file system containing the file.
inode I-node number for the file within its file system.
opens Number of times the file was opened while monitored.
total bytes xfrd Total number of bytes read/written to/from the file.
reads Number of read calls against the file.
read sizes (bytes) The read transfer-size statistics (avg/min/max/sdev), in bytes.
read times (msec) The read response-time statistics (avg/min/max/sdev), in milliseconds.
writes Number of write calls against the file.
write sizes (bytes) The write transfer-size statistics.
write times (msec) The write response-time statistics.
seeks Number of lseek subroutine calls.

Detailed VM Segment Stats Report

Column Description
SEGMENT Internal segment ID.
segtype Type of segment contents.
segment flags Various segment attributes.
volume For persistent segments, the name of the logical volume containing the corresponding file.
inode For persistent segments, the i-node number for the corresponding file.
reads Number of 4096-byte pages read into the segment (that is, paged in).
read times (msec) The read response-time statistics (avg/min/max/sdev), in milliseconds.
read sequences Number of read sequences. A sequence is a string of pages that are read (paged in) consecutively. The number of read sequences is an indicator of the amount of sequential access.
read seq. lengths Statistics describing the lengths of the read sequences, in pages.
writes Number of pages written from the segment (that is, paged out).
write times (msec) Write response time statistics.
write sequences Number of write sequences. A sequence is a string of pages that are written (paged out) consecutively.
write seq.lengths Statistics describing the lengths of the write sequences, in pages.

Detailed Logical/Physical Volume Stats Reports

Column Description
VOLUME Name of the volume.
description Description of the volume. (Describes contents, if discussing a logical volume; describes type, if dealing with a physical volume.)
reads Number of read requests made against the volume.
read sizes (blks) The read transfer-size statistics (avg/min/max/sdev), in units of 512-byte blocks.
read times (msec) The read response-time statistics (avg/min/max/sdev), in milliseconds.
read sequences Number of read sequences. A sequence is a string of 512-byte blocks that are read consecutively and indicate the amount of sequential access.
read seq. lengths Statistics describing the lengths of the read sequences, in blocks.
writes Number of write requests made against the volume.
write sizes (blks) The write transfer-size statistics.
write times (msec) The write-response time statistics.
write sequences Number of write sequences. A sequence is a string of 512-byte blocks that are written consecutively.
write seq. lengths Statistics describing the lengths of the write sequences, in blocks.
seeks Number of seeks that preceded a read or write request; also expressed as a percentage of the total reads and writes that required seeks.
seek dist (blks) Seek distance statistics, in units of 512-byte blocks. In addition to the usual statistics (avg/min/max/sdev), the distance of the initial seek operation (assuming block 0 was the starting position) is reported separately. This seek distance is sometimes very large, so it is reported separately to avoid skewing the other statistics.
seek dist (cyls) (Hard files only.) Seek distance statistics, in units of disk cylinders.
time to next req Statistics (avg/min/max/sdev) describing the length of time, in milliseconds, between consecutive read or write requests to the volume. This column indicates the rate at which the volume is being accessed.
throughput Total volume throughput, in Kilobytes per second.
utilization Fraction of time the volume was busy. The entries in this report are sorted by this field, in decreasing order.

Flags

-i Trace_File Reads the I/O trace data from the specified Trace_File, instead of from the real-time trace process. The filemon report summarizes the I/O activity for the system and period represented by the trace file.
Note
Trace data files are usually written in a circular manner. If the trace data has wrapped around, the chronological beginning and end of the trace may occur in the middle of the file. Use the raw mode of the trcrpt command to rewrite the data sequentially, before invoking the filemon command, as follows:
trcrpt -r file > new.file

For the report to be accurate, the trace file must contain all the hooks required by the filemon command.

The -n option must also be specified.

-n Gennames_File Specifies a Gennames_File for offline trace processing. This file is created by running the gennames command with -f option and redirecting the output to a file, as follows:

gennames -f > file

The -i option must also be specified.

-o File Writes the I/O activity report to the specified File, instead of to the stdout file.
-d Starts the filemon command, but defers tracing until the trcon command has been executed by the user. By default, tracing is started immediately.
-T n Sets the kernel's trace buffer size to n bytes. The default size is 32,000 bytes. The buffer size can be increased to accommodate larger bursts of events, if any. (A typical event record size is 30 bytes.)
Note
The trace driver in the kernel uses double buffering, so in fact there will be two buffers allocated of size n bytes. Also, note that these buffers are pinned in memory, so they are not subject to paging. Large buffers may affect the performance of paging and other I/O.
-P Pins monitor process in memory. The -P flag causes the filemon command's text and data pages to be pinned in memory for the duration of the monitoring period. This flag can be used to ensure that the real-time filemon process is not paged out when running in a memory-constrained environment.
-v Prints extra information in the report. The most significant effect of the -v flag is that all logical files and all segments that were accessed are included in the I/O activity report, instead of only the 20 most active files and segments.
-O Levels Monitors only the specified file system levels. Valid level identifiers are:
lf
Logical file level
vm
Virtual memory level
lv
Logical volume level
pv
Physical volume level
all
Short for lf, vm, lv, pv
The vm, lv, and pv levels are implied by default.
-u Reports on files that were opened prior to the start of the trace daemon. The process ID (PID) and the file descriptor (FD) are substituted for the file name.
Note
Since PIDs and FDs are reusable, it is possible to see different files reported with the same name field.

Examples

  1. To monitor the physical I/O activity of the virtual memory, logical volume, and physical volume levels of the file system, enter:

    filemon

    The filemon command automatically starts the system trace and puts itself in the background. After this command, enter the application programs and system commands to be run at this time, then enter:

    trcstop

    After the trcstop command is issued, the I/O activity report is displayed on standard output (but will probably scroll off the screen). The virtual memory I/O report will be limited to the 20 segments that incurred the most I/O.

  2. To monitor the activity at all file system levels, and write the report to the fmon.out file, enter:

    filemon -o fmon.out -O all

    The filemon command automatically starts the system trace and puts itself in the background. After this command, enter the application programs and system commands to be run at this time, then enter:

    trcstop

    After the trcstop command is issued, the I/O activity report is written to the fmon.out file. All four levels of the file and I/O system (the logical file, virtual memory, logical volume, and physical volume levels) will be monitored. The logical file and virtual memory I/O reports will be limited to the 20 files and segments (respectively) that incurred the most I/O.

  3. To monitor the activity at all file system levels and write a verbose report to the fmon.out file, enter:

    filemon -v -o fmon.out -O all

    The filemon command automatically starts the system trace and puts itself in the background. After this command, enter the application programs and system commands to be run at this time, then enter:

    trcstop

    This example is similar to the previous example, except a verbose report is generated on the fmon.out file. The primary difference is that the filemon command will indicate the steps it is taking to start up the trace, and the summary and detailed reports will include all files and segments that incurred any I/O (there may be many), instead of just the top 20.

  4. To report on I/O activity captured by a previously recorded trace session, enter:

    filemon -i trcfile | pg

    In this example, the filemon command reads file system trace events from the input file trcfile. The input file must already be in raw trace format, as a result of running the trcrpt -r command. Since the trace data is already captured on a file, the filemon command does not put itself in the background to allow application programs to be run. After the entire file is read, an I/O activity report for the virtual memory, logical volume, and physical volume levels will be displayed on standard output (which, in this example, is piped to pg).

  5. To monitor the I/O activity for logical and physical volumes only, while controlling the monitored intervals using the trcon and trcoff commands, enter:

    filemon -d -o fmon.out -O pv,lv

    The filemon command automatically starts the system trace and puts itself in the background. After this command, you can enter the unmonitored application programs and system commands to be run at this time, then enter:

    trcon

    After this command, you can enter the monitored application programs and system commands to be run at this time, then enter:

    trcoff

    After this command, you can enter the unmonitored application programs and system commands to be run at this time, then enter:

    trcon

    After this command, you can enter the monitored application programs and system commands to be run at this time, then enter:

    trcstop

    In this example, the -O flag is used to restrict monitoring to logical and physical volumes only. Only those trace events that are relevant to logical and physical volumes are enabled. Also, as a result of using the -d flag, monitoring is initially deferred until the trcon command is issued. System tracing can be intermittently disabled and reenabled using the trcoff and trcon commands, so that only specific intervals are monitored.

  6. To run filemon in offline mode, run the trace and gennames commands separately, then use the output from those commands as input to the filemon command, as follows:

    trace -a -T 768000 -L 10000000 -o trace.out -j 000,000,001,002,003,005,006,139,102,10C,106,00A,107,
    101,104,10D,15B,12E,130,163,19C,154,3D3,1BA,1BE,1BC,10B,221,1C9,222,228,232,45B

    Run the monitored application programs and system commands, then enter:

    trcstop

    Then format the trace file:

    trcrpt -r trace.out > trace.rpt

    Create the gennames file:

    gennames -f > gennames.out

    Then run filemon with both -i and -n flags:

    filemon -i trace.rpt -n gennames.out -O all

Related Information

The svmon command, trcrpt command, trcstop command.

The lseek subroutine.

Monitoring and Tuning Disk I/O in AIX 5L Version 5.2 Performance Management Guide.

[ Top of Page | Previous Page | Next Page | Contents | Index | Library Home | Legal | Search ]