This document describes how to check for resource bottlenecks and identify the processes that hog them. Resources on a system include memory, cpu, and Input/Output (I/O). This document covers bottlenecks across an entire system. This document does not address the bottlenecks of a particular application or general network problems. The following commands are described:
Note: PAIDE/6000 must be installed in order to use tprof, svmon, netpmon, and filemon. To check if this is installed, run the following command:
lslpp -l perfagent.tools.
To order PAIDE/6000:
This fax also makes reference to the 'vmtune' and 'schedtune' commands. These commands and their source are found in the /usr/samples/kernel directory. They are installed with the bos.adt.samples fileset.
The following section describes memory bottleneck solutions with the following commands: vmstat, svmon, ps
Run the following command:
vmstat 1
Note: System may slow down when 'pi' and 'po' are consistently non-zero.
When processes on the system require more pages of memory than are available in RAM, working pages may be paged out to paging space and then paged in when they are needed again. Accessing a page from paging space is considerably slower than accessing a page directly from RAM. For this reason, constant paging activity can cause system performance degradation.
Note: Memory is over-committed when the 'fr:sr' ratio is high.
An fr:sr ratio of 1:4 means for every one page freed, four pages must be examined. It is difficult to determine a memory constraint based on this ratio alone and what constitutes a "high" ratio is workload/application dependent.
Note: Memory is over-committed to the point of thrashing when 'po*SYS>fr'
The system considers itself to be thrashing when po*SYS > fr (where SYS is a system parameter viewed with the 'schedtune' command). The default value is 0 if a system has 128MB or more. Otherwise,the default is 6. Thrashing occurs when the system spends more time paging than performing work. When this occurs, selected processes may be suspended temporarily, and the system may be noticeably slower.
As root run the following command:
# svmon -Pau 10 | more
Sample Output:
Pid Command Inuse Pin Pgspace 13794 dtwm 1603 1 449 Pid: 13794 Command: dtwm Segid Type Description Inuse Pin Pgspace Address Range b23 pers /dev/hd2:24849 2 0 0 0..1 14a5 pers /dev/hd2:24842 0 0 0 0..2 6179 work lib data 131 0 98 0..891 280a work shared library text 1101 0 10 0..65535 181 work private 287 1 341 0..310:65277..65535 57d5 pers code,/dev/hd2:61722 82 0 0 0..135
This command lists the top ten memory using processes and gives a report about each one. In each process report, look where "Type" = work and "Description" = private. Check how many 4K (4096 byte) pages are used under the "Pgspace" column. This is the minimum number of working pages this segment is using in all of virtual memory. A Pgspace number that grows, but never decreases, may indicate a memory leak. Memory leaks occur when an application fails to deallocate memory.
341 * 4096 = 1,396,736 or 1.4MB of virtual memory
Run the following command:
ps gv | head -n 1; ps gv | egrep -v "RSS" | sort +6b -7 -n -r
The following section describes CPU bottleneck solutions using the following commands: vmstat, tprof, ps
Run the following command:
vmstat 1
Note: System may slow down when processes wait on the run queue.
If the id value is consistently 0%, the cpu is being used 100% of the time.
Look next at the "r" column to see how many threads are placed on the run queue per second. The higher number of threads forced to wait on the run queue, the more system performance will suffer.
To find out how much CPU time a process is using, run the following command as root:
# tprof -x sleep 30
This returns in 30 seconds and creates a file in the current directory called __prof.all.
In 30 seconds, the cpu is checked approximately 3000 times. The "Total" column is the number of times a process was found in the cpu. If one process has 1500 in the "Total" column, this process has taken 1500/3000 or half of the cpu time. The tprof output explains exactly what processes the cpu has been running. The "wait" process runs when no other processes require the cpu and accounts for the amount of idle time on the system.
To find out how much CPU time a process is using, and how much of that time is spent executing network-related code, run the following command as root:
# netpmon -o /tmp/netpmon.out -O cpu -v;sleep 30;trcstop
This returns in 30 seconds and creates a file in the /tmp directory called netpmon.out.
The "CPUTime" indicates the total amount of CPU time for the process. "%CPU" is the percent CPU usage for the process, and "Network CPU%" is the percent of total time that the process spent executing network-related code.
Run the following commands:
ps -ef | head -n 1 ps -ef | egrep -v "UID|0:00|\ 0\ " | sort +3b -4 -n -r
Check the "C" column for a process penalty for recent CPU usage. The maximum value for this column is 120.
ps -e | head -n 1
ps -e | egrep -v "TIME|0:" | sort +2b -3 -n -r
Check the "Time" column for process accumulated CPU time.
ps gu
ps gu | egrep -v "CPU|kproc" | sort +2b -3 -n -r
Check the "%CPU" column for process CPU dependency. The percent CPU is the total CPU time divided by the the total elapsed time since the process was started.
This section describes bottleneck solutions using the following commands: iostat, filemon
Note: High iowait will cause slower performance.
Run the following command:
iostat 5
The time is attributed to iowait when no processes are ready for the cpu but at least one process is waiting on I/O. A high percentage of iowait time indicates that disk I/O is a major contributor to the delay in execution of processes. In general, if system slowness occurs and %iowait is 20% to 25% or higher, investigation of a disk bottleneck is in order.
Note: High tm_act percentage can indicate a disk bottleneck.
When %tm_act or time active for a disk is high, noticeable performance degradation can occur. On some systems, a %tm_act of 35% or higher for one disk can cause noticeably slow performance.
To find out what files, logical volumes, and disks are most active, run the following command as root:
# filemon -u -O all -o /tmp/fmon.out; sleep 30;trcstop
In 30 seconds, a report is created in /tmp/fmon.out.
The Most Active Segments report lists the most active files by file system and inode. The mount point of the file system and inode of the file can be used with the ncheck command to identify unknown files:
# ncheck -i <inode> <mount point>
This report is useful in determining if the activity is to a filesystem (segtype = persistent), the JFS log (segtype = log), or to paging space (segtype = working).
By examining the reads and read sequences counts, you can determine if the access is sequential or random. As the read sequences count approaches the reads count, file access is more random. The same applies to the writes and write sequences.
Performance Tools
cpu_state -l
Shows what state each processor is currently in (enabled, disabled, or unavailable).
ps -m -o THREAD
The BND column will indicate the processor number to which a process/thread is bound to (if it is bound).
pstat -A
The CPUID column will indicate the processor number to which a process/thread is bound to.
sar -P ALL
Load on all the processors.
vmstat
Shows "kthr"(kernel threads)
netpmon -t
Prints CPU reports on a per-thread basis.
cpu_state -l
sar -P ALL
ps -m -o THREAD
pstat -A
The Block I/O Buffer Cache (KBUFFERS) is only used when directly accessing a block device, such as /dev/hdisk0. Normal access through the Journaled File System (JFS) is managed by the Virtual Memory Manager (VMM) and does not use the traditional method for caching the data blocks. Any I/O operations to raw logical volumes or physical volumes does not use the Block I/O Buffer Cache.
Users of AIX occasionally encounter long interactive-application response times when other applications in the system are running large writes to disk. Because most writes are asynchronous, FIFO I/O queues of several megabytes may build up, and take several seconds to complete. The performance of an interactive process is severely impacted if every disk read spends several seconds working through the queue. I/O pacing limits the number of outstanding I/O requests against a file. When a process tries to write to a file whose queue is at the high-water mark (should be a multiple of 4 plus 1), the process is suspended until enough I/Os have completed to bring the queue for that file to the low-water mark. The delta between the high and low water marks should be kept small.
To configure I/O pacing on the system via 'smit', enter the following at the command line as root:
# smitty chgsys
Async I/O is performed in the background and does not block the user process. This improves performance because I/O operations and application processing can run concurrently. However, applications must be specifically written to take advantage of aysnc I/O that is managed by the aio daemons running on the system.
To configure Async I/O the system via 'smit', enter the following at the command line as root:
# smitty aio
Consult Line Performance Analysis - The AIX Support Family offers a system analysis with tuning recommendations. For more information (U.S.) call 1-800-CALL-AIX , other countries call your local support structure.
Performance Tuning Guide (SC23-2365) - This IBM publication covers performance monitoring and tuning of AIX systems. Order through your local IBM representative or (U.S.) by calling IBM Publications at 1-800-879-2755.
For detailed system usage on a per process basis, a free utlity called UTLD can be obtained by anonymous ftp from ftp.software.ibm.com in the /aix/tools/perftools/utld directory. For more information see the README file /usr/lpp/utld after installation of the utld.obj fileset.