Monitors the performance of the file system, and reports the I/O activity on behalf of logical files, virtual memory segments, logical volumes, and physical volumes.
filemon [ -d ] [ -i Trace_File -n Gennames_File] [ -o File] [ -O Levels] [ -P ] [ -T n] [ -u ] [ -v ]
The filemon command monitors a trace of file system and I/O system events, and reports on the file and I/O access performance during that period.
In its normal mode, the filemon command runs in the background while one or more application programs or system commands are being executed and monitored. The filemon command automatically starts and monitors a trace of the program's file system and I/O events in real time. By default, the trace is started immediately; optionally, tracing may be deferred until the user issues a trcon command. The user can issue trcoff and trcon commands while the filemon command is running in order to turn off and on monitoring, as desired. When tracing is stopped by a trcstop command, the filemon command generates an I/O activity report and exits.
The filemon command can also process a trace file that has been previously recorded by the trace facility. The file and I/O activity report will be based on the events recorded in that file.
To provide a more complete understanding of file system performance for an application, the filemon command monitors file and I/O activity at four levels:
| Logical file system | The filemon command monitors logical I/O operations on logical files. The monitored operations include all read, write, open, and lseek system calls, which may or may not result in actual physical I/O, depending on whether or not the files are already buffered in memory. I/O statistics are kept on a per-file basis. |
| Virtual memory system | The filemon command monitors physical I/O operations (that is, paging) between segments and their images on disk. I/O statistics are kept on a per-segment basis. |
| Logical volumes | The filemon command monitors I/O operations on logical volumes. I/O statistics are kept on a per-logical-volume basis. |
| Physical volumes | The filemon command monitors I/O operations on physical volumes. At this level, physical resource utilizations are obtained. I/O statistics are kept on a per-physical-volume basis. |
Any combination of the four levels can be monitored, as specified by the command line flags. By default, the filemon command only monitors I/O operations at the virtual memory, logical volume, and physical volume levels. These levels are all concerned with requests for real disk I/O.
The filemon command writes its report to standard output or to a specified file. The report begins with a summary of the I/O activity for each of the levels being monitored and ends with detailed I/O activity statistics for each of the levels being monitored. Summary and detailed report contents are described in the Reports section.
The -u flag is used to generate reports on files opened prior to the start of the trace daemon. Some of this data can be useful, but much of it applies to daemons and other unrelated activity. This background information can be overwhelming, especially on large systems. If the /unix file and the running kernel are not the same, then the kernel addresses will be incorrect, causing the filemon command to exit. When using the filemon command from within a shell script, allow for a slight delay prior to viewing the contents of the filemon output file. The filemon command may take a few seconds to produce this report.
The filemon command obtains raw I/O performance data using the system trace facility. Currently, the trace facility only supports one output stream. Consequently, only one filemon or trace process can be active at a time. If another filemon or trace process is already running, the filemon command responds with the message:
/dev/systrace: Device busy
While monitoring very I/O-intensive applications, the filemon command may not be able to consume trace events as fast as they are produced in real time. When that happens, the error message:
Trace kernel buffers overflowed, N missed entries
will be displayed on stderr, indicating how many trace events were lost while the trace buffers were full. The filemon command will continue monitoring I/O activity, but the accuracy of the report will be diminished to some unknown degree. One way to prevent overflow is to monitor fewer levels of the file and I/O subsystems: the number of trace events generated is proportional to the number of levels monitored. Additionally, the trace buffer size can be increased using the -T option, to accommodate larger bursts of trace events before overflow. Remember that increasing the trace buffer size will result in more pinned memory, and therefore may effect I/O and paging behavior.
In memory-constrained environments (where demand for memory exceeds supply), the -P option can be used to pin the text and data pages of the real-time filemon process in memory so the pages cannot be swapped out. If the -P option is not used, allowing the filemon process to be swapped out, the progress of the filemon command may be delayed to the point where it cannot process trace events fast enough. This situation leads to trace buffer overflow as described above. Of course, pinning this process takes memory away from the application (although the filemon command is not a large program, its process image can consume up to 500KB).
Before using the filemon command to process an existing trace data file, you must use the -r option of the trcrpt command to rewrite the trace data sequentially to a new file. Otherwise, the filemon command produces the following error message, and then exits:
error: run 'trcrpt -r' on logfile first
The -i Trace_File and -n Gennames_File flags allow for offline processing by filemon of trace data files created by the trace command. Both flags must be supplied if either is present. These flags are useful when it is necessary to postprocess a trace file from a remote machine or perform the trace data collection at one time and postprocess it at another time. The flags are also useful when system load is high and trace hooks are being missed by filemon.
The gennames file (containing filesystem information) must be used from the machine that the trace came from. Also, it is wise to run gennames at close to the same time that the system trace file is created, so that the system configuration is the same for both.
Trace hooks relevant to filemon must be collected by the trace command and are specified by the trace -j flag. The relevant trace hooks are listed when filemon is invoked with the -v flag. The gennames command with -foption is then executed, with its output saved in Gennames_File to collect additional information for filemon. The -f option is used with the gennames command to collect the device information for physical and logical volumes. It is also used to get the virtual file system information used by offline filemon. Once the trace command has been executed, trcrpt -r must be run on the trace logfile and redirected to another file. Then this file and the Gennames_File may be provided to filemon.
Each report generated by the filemon command has a header that identifies the date, the machine ID, and the length of the monitoring period, in seconds. The CPU utilization during the monitoring period is also reported.
Next, summary reports are generated for each of the file system levels being monitored. By default, the logical file and virtual memory reports are limited to the 20 most active files and segments, respectively, as measured by the total amount of data transferred. If the -v flag has been specified, activity for all files and segments is reported. There is one row for each reported file, segment, or volume. The columns in each row for the four summary reports are described in the following lists:
| Column | Description |
| #MBS | Total number of megabytes transferred to/from segment. The rows are sorted by this field, in decreasing order. |
| #rpgs | Number of 4096-byte pages read into segment from disk (that is, page). |
| #wpgs | Number of 4096-byte pages written from segment to disk (page out). |
| segid | Internal ID of segment. |
| segtype | Type of segment: working segment, persistent segment (local file), client segment (remote file), page table segment, system segment, or special persistent segments containing file system data (log, root directory, .inode, .inodemap, .inodex, .inodexmap, .indirect, .diskmap). |
| volume:inode | For persistent segments, name of volume that contains the associated
file, and the file's inode number. This field can be used to associate a persistent
segment with its corresponding file, shown in the file I/O reports. This field
is blank for non-persistent segments.
Note
The virtual memory analysis
tool, svmon can be used
to display more information about a segment, given its segment ID (segid),
as follows:
svmon -S <segid> |
Finally, detailed reports are generated for each of the file system levels being monitored. By default, the logical file and virtual memory reports are limited to the 20 most active files and segments, respectively, as measured by the total amount of data transferred. If the -v flag is specified, activity for all files and segments is reported. There is one entry for each reported file, segment, or volume.
Some of the fields report a single value, others report statistics that characterize a distribution of many values. For example, response time statistics are kept for all read or write requests that were monitored. The average, minimum, and maximum response times are reported, as well as the standard deviation of the response times. The standard deviation is used to show how much the individual response times deviated from the average. Roughly two-thirds of the sampled response times are between average - standard deviation and average + standard deviation. If the distribution of response times is scattered over a large range, the standard deviation will be large compared to the average response time. The four detailed reports are described in the following lists:
| Column | Description |
| FILE | Name of the file. The full path name is given, if possible. |
| volume | Name of the logical volume/file system containing the file. |
| inode | I-node number for the file within its file system. |
| opens | Number of times the file was opened while monitored. |
| total bytes xfrd | Total number of bytes read/written to/from the file. |
| reads | Number of read calls against the file. |
| read sizes (bytes) | The read transfer-size statistics (avg/min/max/sdev), in bytes. |
| read times (msec) | The read response-time statistics (avg/min/max/sdev), in milliseconds. |
| writes | Number of write calls against the file. |
| write sizes (bytes) | The write transfer-size statistics. |
| write times (msec) | The write response-time statistics. |
| seeks | Number of lseek subroutine calls. |
| -i Trace_File | Reads the I/O trace data from the specified Trace_File, instead of from the real-time trace process. The filemon report summarizes the I/O activity for the system and period
represented by the trace file.
Note
Trace data files are usually
written in a circular manner. If the trace data has wrapped around, the chronological
beginning and end of the trace may occur in the middle of the file. Use the
raw mode of the trcrpt command to rewrite the data sequentially,
before invoking the filemon command, as follows:
trcrpt -r file > new.file For the report to be accurate, the trace file must contain all the hooks required by the filemon command. The -n option must also be specified. |
| -n Gennames_File | Specifies a Gennames_File for offline trace
processing. This file is created by running the gennames command with -f option and redirecting the output to a file, as follows:
gennames -f > file The -i option must also be specified. |
| -o File | Writes the I/O activity report to the specified File, instead of to the stdout file. |
| -d | Starts the filemon command, but defers tracing until the trcon command has been executed by the user. By default, tracing is started immediately. |
| -T n | Sets the kernel's trace buffer size to n bytes.
The default size is 32,000 bytes. The buffer size can be increased to accommodate
larger bursts of events, if any. (A typical event record size is 30 bytes.)
Note
The trace driver in the kernel uses double buffering, so
in fact there will be two buffers allocated of size n
bytes. Also, note that these buffers are pinned in memory, so they are not
subject to paging. Large buffers may affect the performance of paging and
other I/O. |
| -P | Pins monitor process in memory. The -P flag causes the filemon command's text and data pages to be pinned in memory for the duration of the monitoring period. This flag can be used to ensure that the real-time filemon process is not paged out when running in a memory-constrained environment. |
| -v | Prints extra information in the report. The most significant effect of the -v flag is that all logical files and all segments that were accessed are included in the I/O activity report, instead of only the 20 most active files and segments. |
| -O Levels | Monitors only the specified file system levels. Valid level identifiers
are:
|
| -u | Reports on files that were opened prior to the start of the trace daemon. The process ID (PID) and the file descriptor
(FD) are substituted for the file name.
Note
Since PIDs and FDs
are reusable, it is possible to see different files reported with the same
name field. |
filemon
The filemon command automatically starts the system trace and puts itself in the background. After this command, enter the application programs and system commands to be run at this time, then enter:
trcstop
After the trcstop command is issued, the I/O activity report is displayed on standard output (but will probably scroll off the screen). The virtual memory I/O report will be limited to the 20 segments that incurred the most I/O.
filemon -o fmon.out -O all
The filemon command automatically starts the system trace and puts itself in the background. After this command, enter the application programs and system commands to be run at this time, then enter:
trcstop
After the trcstop command is issued, the I/O activity report is written to the fmon.out file. All four levels of the file and I/O system (the logical file, virtual memory, logical volume, and physical volume levels) will be monitored. The logical file and virtual memory I/O reports will be limited to the 20 files and segments (respectively) that incurred the most I/O.
filemon -v -o fmon.out -O all
The filemon command automatically starts the system trace and puts itself in the background. After this command, enter the application programs and system commands to be run at this time, then enter:
trcstop
This example is similar to the previous example, except a verbose report is generated on the fmon.out file. The primary difference is that the filemon command will indicate the steps it is taking to start up the trace, and the summary and detailed reports will include all files and segments that incurred any I/O (there may be many), instead of just the top 20.
filemon -i trcfile | pg
In this example, the filemon command reads file system trace events from the input file trcfile. The input file must already be in raw trace format, as a result of running the trcrpt -r command. Since the trace data is already captured on a file, the filemon command does not put itself in the background to allow application programs to be run. After the entire file is read, an I/O activity report for the virtual memory, logical volume, and physical volume levels will be displayed on standard output (which, in this example, is piped to pg).
filemon -d -o fmon.out -O pv,lv
The filemon command automatically starts the system trace and puts itself in the background. After this command, you can enter the unmonitored application programs and system commands to be run at this time, then enter:
trcon
After this command, you can enter the monitored application programs and system commands to be run at this time, then enter:
trcoff
After this command, you can enter the unmonitored application programs and system commands to be run at this time, then enter:
trcon
After this command, you can enter the monitored application programs and system commands to be run at this time, then enter:
trcstop
In this example, the -O flag is used to restrict monitoring to logical and physical volumes only. Only those trace events that are relevant to logical and physical volumes are enabled. Also, as a result of using the -d flag, monitoring is initially deferred until the trcon command is issued. System tracing can be intermittently disabled and reenabled using the trcoff and trcon commands, so that only specific intervals are monitored.
trace -a -T 768000 -L 10000000 -o trace.out -j 000,000,001,002,003,005,006,139,102,10C,106,00A,107, 101,104,10D,15B,12E,130,163,19C,154,3D3,1BA,1BE,1BC,10B,221,1C9,222,228,232,45B
Run the monitored application programs and system commands, then enter:
trcstop
Then format the trace file:
trcrpt -r trace.out > trace.rpt
Create the gennames file:
gennames -f > gennames.out
Then run filemon with both -i and -n flags:
filemon -i trace.rpt -n gennames.out -O all
The svmon command, trcrpt command, trcstop command.
The lseek subroutine.
Monitoring and Tuning Disk I/O in AIX 5L Version 5.2 Performance Management Guide.