This chapter provides an overview to the Performance Toolbox for AIX (PTX) and Performance Aide for AIX (PAIDE) products.
Anyone faced with the task of keeping a computer system well-tuned and capable of performing as expected recognizes the following areas as essential for success:
- Load Monitoring
- Resource load must be monitored so performance problems can be detected as they occur or (preferably) predicted well before they do.
- Analysis and Control
- Once a performance problem is encountered, the proper tools must be selected and applied so that the nature of the problem can be understood and corrective action taken.
- Capacity Planning
- Long-term capacity requirements must be analyzed so sufficient resources can be acquired well before they are required.
This book describes the Performance Toolbox for AIX, which is designed to help a system administrator do exactly this. The Performance Toolbox for AIX is divided into two components:
The two components constitute the server (Agent) and the client (Manager) sides of a set of performance management tools, which allow performance monitoring and analysis in a networked environment.
The four main programs of the Manager component are all X Window System based programs developed with the OSF/Motif Toolkit. One program, xmperf, is at the same time a graphical monitor, a tool for analyzing system load, and an umbrella for other tools, performance related or not. Another monitoring program is 3dmon, which allows the monitoring of a large number of statistics in a single window. The program exmon is designed to work with the filtd daemon. It monitors alarms generated by filtd. The fourth main program is the azizo program to analyze recordings of performance data.
The main program in the Agent component is the daemon xmservd, which acts as a networked supplier of performance statistics and, optionally, as a supplier of performance statistics to Simple Network Management Protocol (SNMP) managers.
Both components include application programming interfaces for application controlled access and supply of performance statistics.
The client/server environment allows any program, whether part of Performance Toolbox for AIX or custom developed applications, to monitor the local host as well as multiple remote hosts. This ability is fully explored in the Manager component program xmperf whose "monitors" are graphical windows, referred to as consoles, which can be customized on the fly or kept as pre-configured consoles that can be invoked with a few mouse clicks. Consoles can be generic so the actual resource to monitor, whether a remote host, a disk drive, or a LAN interface, is chosen when the console is opened. Consoles can be told to do a recording to disk files of the data they monitor and such recordings can be played back with xmperf and analyzed with the azizo program.
One of the things that makes xmperf unique is that it is not hardcoded to monitor a fixed set of resources. It is dynamic in the sense that a system administrator can customize it to focus on exactly the resources that are critical for each host that must be monitored.
An implementation of the Application Response Management (ARM) specification allows applications to be instrumented so that the activity and response times of applications can be monitored. In addition, the raw response time from any host running the PTX agent to any IP capable host in the network can be monitored.
Another feature is the Agent filter called filtd, which allows you to easily combine existing "raw" statistics into new statistics that make more sense in your environment. This can be done by entering simple expressions in a configuration file and requires no programming.
By providing an umbrella for tools that can be used to analyze performance data and control system resources, the Manager program xmperf assists the system administrator in keeping track of available tools and in applying them in appropriate ways. This is done through a customizable menu interface. Tools can be added to menus, either with fixed sets of command line arguments to match specific situations or such that the system administrator has an easy way to remember and enter command line arguments in a dialog window. The menus of xmperf are preconfigured to include most of the performance tools shipped as part of the tools option of the Agent component.
Properly customized, xmperf becomes an indispensable repository for tools to analyze and control an AIX system. In addition, the ability to record load scenarios and play them back in graphical windows at any desired speed gives new and improved ways of analyzing a performance problem.
Outstanding features for analyzing a recording of performance data are provided by the azizo program and its support programs. Recordings can be produced from the monitoring programs xmperf and 3dmon during monitoring, or can be created by the xmservd daemon. The xmservd daemon allows for recording with a minimum of overhead. This makes constant recording possible so that you can analyze performance problems after they occurred. The 3dplay program is provided to play back recordings created by 3dmon in the same style in which the data was originally displayed.
Finally, using the Agent component filter filtd, you can define conditions that, when met, could trigger any action you deem appropriate, including alerting yourself and/or initiating corrective action without human intervention. This facility is entirely configurable so that alarms and actions can be customized to your installation.
If you can make your system simulate a future load scenario, xmperf can be used to visualize the resulting performance of your system. By simulating the load scenario on systems with more resources, such as more memory or more disks, the result of increasing the resources can be demonstrated.
The xmservd data-supplier daemon can provide consumers of performance statistics with a stream of data. Frequency and contents of each packet of performance data are determined by the consumer program. Any consumer program can access performance data from the local host and one or more remote hosts; any data-supplier daemon can supply data to multiple hosts.
In addition to its ability to monitor across a network, PTX also allows for monitoring the response time to and from nodes in the network itself.
Each component comes with its own application programming interface (API). In addition, an API is available for instrumentation of application programs:
- Agent API
- Called the System Performance Measurement Interface (SPMI) API . It allows an application program to register custom performance statistics about its own performance or that of some other system component. Once registered, the custom statistics become available to any consumer of statistics, local or remote. Programs that supply custom statistics are called dynamic data-supplier programs.
The Agent API also permits applications to access statistics on the local system without using the network interface. Such applications are called local data-consumer programs.
- Manager API
- Called the Remote Statistics Interface (RSi) API . Allows an application program to access statistics from remote nodes (or the local host) through a network interface.
- Application Monitoring API
- Called the Application Response Management (ARM) , this API permits application program to be instrumented in such a way that the application activity and response time can be monitored from any of the PTX manager programs.
By entering a single keyword in a configuration file, the data-supplier daemon can be told to export all its statistics to a local snmpd SNMP agent. Users of an SNMP manager such as IBM NetView see the exported statistical data as an extension of the set of data already available from snmpd.
Note: The SNMP multiplex interface is only available on IBM RISC System/6000 Agents.
Throughout this book, the terms statistic and metric are used to describe a probe in or instrumentation of a component of the operating system or, in some cases, application programs. The probe is normally defined and updated by the system, regardless of the presence of Performance Toolbox for AIX. In the current version, a probe is always represented by a data field that is either incremented each time a certain event occurs (a counter) or represents a quantity, such as the amount of free disk or memory.
In this book, the terms statistic and metric are synonymous. Usually metric is used in connection with the azizo program and related programs. The term statistic is the preferred term in describing the APIs and other programs.
Finally, the term value is used to refer to a statistic (metric) when included in a monitoring "device" in the programs xmperf and 3dmon.
The Agent component of the Performance Toolbox for AIX is identical to the Performance Aide for AIX licensed product server option. It has the following main components:
- xmservd
- The data-supplier daemon, which permits a system where this daemon runs to supply performance statistics to data-consumer programs on the local or remote hosts. This daemon also provides the interface to SNMP.
Note: The interface to SNMP is available only on RS/6000 Agents.- xmscheck
- A program that lets you pre-check the xmservd recording configuration file. This program is very useful when you want to start and stop xmservd recording at predetermined times.
- filtd
- A daemon that can be used to do data reduction of existing statistics and to define alarm conditions and triggering of alarms.
- xmpeek
- A program that allows you to display the status of xmservd on the local or a remote host and to list all available statistics from the daemon.
- iphosts
- A program to initiate monitoring of Internet Protocol performance by specifying which hosts to monitor. Accepts a list of hosts from the command line or from a file.
- armtoleg
- A program that can convert a pre-existing Application Response Management (ARM) library into an ARM library that can be accessed concurrently with the ARM library shipped with PTX. Only required and available on AIX systems.
- SpmiArmd
- A daemon that collects Application Response Management (ARM) data and interfaces to the Spmi library code to allow monitoring of ARM metrics from any PTX manager program.
- SpmiResp
- A daemon that polls for IP response times for selected hosts and interfaces to the Spmi library code to allow monitoring of IP response time metrics from any PTX manager program.
- Application Response Management API and Libraries
- A header file and two libraries support the PTX implementation of ARM. The implementation allows for coexistence and simultaneous use of the PTX ARM library and one previously installed ARM library.
- System Performance Measurement Interface API and Library
- Header files and a library to allow you to develop your own data-supplier and local data-consumer programs.
- Sample Programs
- Sample dynamic data-supplier and data-consumer programs that illustrate the use of the API.
- PTX Agents for selected non-IBM platforms
- Compressed tar files provide full agent support on multiple HP-UX and Solaris Versions on a variety of HP and Sun systems.
The Manager component of the Performance Toolbox for AIX has the following main components:
- xmperf
- The main user interface program providing graphical display of local and remote performance information and a menu interface to commands of your choice.
- 3dmon
- A program that can monitor up to 576 statistics simultaneously and display the statistics in a 3-dimensional graph.
- 3dplay
- A program to play 3dmon recordings back in a 3dmon-like view.
- chmon
- Supplied as an executable as well as in source form, this program allows monitoring of vital statistics from a character terminal.
- exmon
- The program that allows monitoring of alarms generated by the filtd daemon running on remote hosts.
- azizo
- A program that allows you to analyze any recording of performance data. It lets you zoom-in on sections of the recording and provides graphical as well as tabular views of the entire recording or zoomed-in parts of it.
- ptxtab
- A program that can format statset date from recording files for printed output.
- ptxmerge
- This program allows you to merge up to 10 recording files into one. For example, you could merge xmservd recordings from the client and server sides of an application into one file to better correlate the performance impact of the application on the two sides.
- ptxsplit
- In cases where recording files are too large to analyze as one file, this program allows you to split the file into multiple smaller files for better overview and faster analysis.
- ptxrlog
- A program to create recordings in ASCII or binary format.
- ptxls
- A program to list the control information of a recording file, including a list of the statistics defined in the file.
- a2ptx
- The a2ptx program can generate recordings from ASCII files in a format as produced by the ptxtab or ptxrlog programs or the Performance Toolbox for AIX SpmiLogger sample program. The generated recording can then be played back by xmperf or analyzed with azizo.
- ptxconv
- The format of recordings has changed between Versions of the Performance Toolbox for AIX. As a convenience to users of multiple versions of the Performance Toolbox for AIX, this program converts recording files between the formats of the different versions.
- ptx2stat
- Converts hotset data collected in a recording file to a format that resembles the recording format for statsets. Permits postprocessing of hotset data with the programs that allow playback and manipulation of recordings.
- ptxhottab
- A program that can format and print hotset information collected in recording files.
- Remote Statistics Interface API
- Header file to allow you to develop your own data-consumer programs.
- Sample Programs
- Sample data-consumer programs that illustrate the use of the API.