Red Hat Training

A Red Hat training course is available for Red Hat Enterprise Linux

8.2. Monitoring and Diagnosing Performance Problems

Red Hat Enterprise Linux 7 provides a number of tools that are useful for monitoring system performance and diagnosing performance problems related to I/O and file systems and their configuration. This section outlines the available tools and gives examples of how to use them to monitor and diagnose I/O and file system related performance issues.

8.2.1. Monitoring System Performance with vmstat

Vmstat reports on processes, memory, paging, block I/O, interrupts, and CPU activity across the entire system. It can help administrators determine whether the I/O subsystem is responsible for any performance issues.
The information most relevant to I/O performance is in the following columns:
si
Swap in, or reads from swap space, in KB.
so
Swap out, or writes to swap space, in KB.
bi
Block in, or block write operations, in KB.
bo
Block out, or block read operations, in KB.
wa
The portion of the queue that is waiting for I/O operations to complete.
Swap in and swap out are particularly useful when your swap space and your data are on the same device, and as indicators of memory usage.
Additionally, the free, buff, and cache columns can help identify write-back frequency. A sudden drop in cache values and an increase in free values indicates that write-back and page cache invalidation has begun.
If analysis with vmstat shows that the I/O subsystem is responsible for reduced performance, administrators can use iostat to determine the responsible I/O device.
vmstat is provided by the procps-ng package. For detailed information about using vmstat, see the man page: 
$ man vmstat

8.2.2. Monitoring I/O Performance with iostat

Iostat is provided by the sysstat package. It reports on I/O device load in your system. If analysis with vmstat shows that the I/O subsystem is responsible for reduced performance, you can use iostat to determine the I/O device responsible.
You can focus the output of iostat reports on a specific device by using the parameters defined in the iostat man page: 
$ man iostat

8.2.2.1. Detailed I/O Analysis with blktrace

Blktrace provides detailed information about how time is spent in the I/O subsystem. The companion utility blkparse reads the raw output from blktrace and produces a human readable summary of input and output operations recorded by blktrace.
For more detailed information about this tool, see the blktrace(8) and blkparse(1) man pages: 
$ man blktrace
$ man blkparse

8.2.2.2. Analyzing blktrace Output with btt

The btt utility is provided as part of the blktrace package. It analyzes blktrace output and displays the amount of time that data spends in each area of the I/O stack, making it easier to spot bottlenecks in the I/O subsystem.
Some of the important events tracked by the blktrace mechanism and analyzed by btt are:
  • Queuing of the I/O event (Q)
  • Dispatch of the I/O to the driver event (D)
  • Completion of I/O event (C)
You can include or exclude factors involved with I/O performance issues by examining combinations of events.
To inspect the timing of sub-portions of each I/O device, look at the timing between captured blktrace events for the I/O device. For example, the following command reports the total amount of time spent in the lower part of the kernel I/O stack (Q2C), which includes scheduler, driver, and hardware layers, as an average under await time: 
$ iostat -x

[...]
Device:         await r_await w_await
vda             16.75    0.97  162.05
dm-0            30.18    1.13  223.45
dm-1             0.14    0.14    0.00
[...]
If the device takes a long time to service a request (D2C), the device may be overloaded, or the workload sent to the device may be sub-optimal. If block I/O is queued for a long time before being dispatched to the storage device (Q2G), it may indicate that the storage in use is unable to serve the I/O load. For example, a LUN queue full condition has been reached and is preventing the I/O from being dispatched to the storage device.
Looking at the timing across adjacent I/O can provide insight into some types of bottleneck situations. For example, if btt shows that the time between requests being sent to the block layer (Q2Q) is larger than the total time that requests spent in the block layer (Q2C), this indicates that there is idle time between I/O requests and the I/O subsystem may not be responsible for performance issues.
Comparing Q2C values across adjacent I/O can show the amount of variability in storage service time. The values can be either:
  • fairly consistent with a small range, or
  • highly variable in the distribution range, which indicates a possible storage device side congestion issue.
For more detailed information about this tool, see the btt(1) man page: 
$ man btt

8.2.2.3. Analyzing blktrace Output with iowatcher

The iowatcher tool can use blktrace output to graph I/O over time. It focuses on the Logical Block Address (LBA) of disk I/O, throughput in megabytes per second, the number of seeks per second, and I/O operations per second. This can help to identify when you are hitting the operations-per-second limit of a device.
For more detailed information about this tool, see the iowatcher(1) man page.

8.2.3. Storage Monitoring with SystemTap

The Red Hat Enterprise Linux 7 SystemTap Beginners Guide includes several sample scripts that are useful for profiling and monitoring storage performance.
The following SystemTap example scripts relate to storage performance and may be useful in diagnosing storage or file system performance problems. By default they are installed to the /usr/share/doc/systemtap-client/examples/io directory.
disktop.stp
Checks the status of reading/writing disk every 5 seconds and outputs the top ten entries during that period.
iotime.stp
Prints the amount of time spent on read and write operations, and the number of bytes read and written.
traceio.stp
Prints the top ten executables based on cumulative I/O traffic observed, every second.
traceio2.stp
Prints the executable name and process identifier as reads and writes to the specified device occur.
inodewatch.stp
Prints the executable name and process identifier each time a read or write occurs to the specified inode on the specified major/minor device.
inodewatch2.stp
Prints the executable name, process identifier, and attributes each time the attributes are changed on the specified inode on the specified major/minor device.