8.2.2. Monitoring I/O Performance with iostat
184.108.40.206. Detailed I/O Analysis with blktrace
Blktrace provides detailed information about how time is spent in the I/O subsystem. The companion utility blkparse reads the raw output from blktrace and produces a human readable summary of input and output operations recorded by blktrace.
For more detailed information about this tool, see the blktrace(8) and blkparse(1) man pages:
$ man blktrace
$ man blkparse
220.127.116.11. Analyzing blktrace Output with btt
The btt utility is provided as part of the blktrace package. It analyzes blktrace output and displays the amount of time that data spends in each area of the I/O stack, making it easier to spot bottlenecks in the I/O subsystem.
Some of the important events tracked by the
blktrace mechanism and analyzed by btt are:
Queuing of the I/O event (
Dispatch of the I/O to the driver event (
Completion of I/O event (
You can include or exclude factors involved with I/O performance issues by examining combinations of events.
To inspect the timing of sub-portions of each I/O device, look at the timing between captured
blktrace events for the I/O device. For example, the following command reports the total amount of time spent in the lower part of the kernel I/O stack (
Q2C), which includes scheduler, driver, and hardware layers, as an average under await time:
Device: await r_await w_await
vda 16.75 0.97 162.05
dm-0 30.18 1.13 223.45
dm-1 0.14 0.14 0.00
If the device takes a long time to service a request (
D2C), the device may be overloaded, or the workload sent to the device may be sub-optimal. If block I/O is queued for a long time before being dispatched to the storage device (
Q2G), it may indicate that the storage in use is unable to serve the I/O load. For example, a LUN queue full condition has been reached and is preventing the I/O from being dispatched to the storage device.
Looking at the timing across adjacent I/O can provide insight into some types of bottleneck situations. For example, if btt shows that the time between requests being sent to the block layer (
Q2Q) is larger than the total time that requests spent in the block layer (
Q2C), this indicates that there is idle time between I/O requests and the I/O subsystem may not be responsible for performance issues.
Q2C values across adjacent I/O can show the amount of variability in storage service time. The values can be either:
fairly consistent with a small range, or
highly variable in the distribution range, which indicates a possible storage device side congestion issue.
For more detailed information about this tool, see the btt(1) man page:
18.104.22.168. Analyzing blktrace Output with seekwatcher
The seekwatcher tool can use blktrace output to graph I/O over time. It focuses on the Logical Block Address (LBA) of disk I/O, throughput in megabytes per second, the number of seeks per second, and I/O operations per second. This can help to identify when you are hitting the operations-per-second limit of a device.
For more detailed information about this tool, see the man page:
$ man seekwatcher