Red Hat Training

A Red Hat training course is available for Red Hat Enterprise Linux

2.3. diskdevstat and netdevstat

diskdevstat and netdevstat are SystemTap tools that collect detailed information about the disk activity and network activity of all applications running on a system. These tools were inspired by PowerTOP, which shows the number of CPU wakeups by every application per second (refer to Section 2.2, “PowerTOP”). The statistics that these tools collect allow you to identify applications that waste power with many small I/O operations rather than fewer, larger operations. Other monitoring tools that measure only transfer speeds do not help to identify this type of usage.
Install these tools with SystemTap with the following command as root: 
yum install systemtap tuned-utils kernel-debuginfo
Run the tools with the command: 
diskdevstat
or the command: 
netdevstat
Both commands can take up to three parameters, as follows: 
diskdevstat update_interval total_duration display_histogram
netdevstat update_interval total_duration display_histogram
update_interval
The time in seconds between updates of the display. Default: 5
total_duration
The time in seconds for the whole run. Default: 86400 (1 day)
display_histogram
Flag whether to histogram for all the collected data at the end of the run.
The output of the diskdevstat command resembles that of PowerTOP. See the example.

Example 2.1. An Output of the diskdevstat Command

Here is sample output from a longer diskdevstat run: 
PID  UID  DEV WRITE_CNT WRITE_MIN WRITE_MAX WRITE_AVG READ_CNT READ_MIN READ_MAX READ_AVG COMMAND
2789  2903 sda1     854     0.000   120.000    39.836      0     0.000     0.000     0.000 plasma
5494     0 sda1       0     0.000     0.000     0.000    758     0.000     0.012     0.000 0logwatch
5520     0 sda1       0     0.000     0.000     0.000    140     0.000     0.009     0.000 perl
5549     0 sda1       0     0.000     0.000     0.000    140     0.000     0.009     0.000 perl
5585     0 sda1       0     0.000     0.000     0.000    108     0.001     0.002     0.000 perl
2573     0 sda1      63     0.033  3600.015   515.226      0     0.000     0.000     0.000 auditd
5429     0 sda1       0     0.000     0.000     0.000     62     0.009     0.009     0.000 crond
5379     0 sda1       0     0.000     0.000     0.000     62     0.008     0.008     0.000 crond
5473     0 sda1       0     0.000     0.000     0.000     62     0.008     0.008     0.000 crond
5415     0 sda1       0     0.000     0.000     0.000     62     0.008     0.008     0.000 crond
5433     0 sda1       0     0.000     0.000     0.000     62     0.008     0.008     0.000 crond
5425     0 sda1       0     0.000     0.000     0.000     62     0.007     0.007     0.000 crond
5375     0 sda1       0     0.000     0.000     0.000     62     0.008     0.008     0.000 crond
5477     0 sda1       0     0.000     0.000     0.000     62     0.007     0.007     0.000 crond
5469     0 sda1       0     0.000     0.000     0.000     62     0.007     0.007     0.000 crond
5419     0 sda1       0     0.000     0.000     0.000     62     0.008     0.008     0.000 crond
5481     0 sda1       0     0.000     0.000     0.000     61     0.000     0.001     0.000 crond
5355     0 sda1       0     0.000     0.000     0.000     37     0.000     0.014     0.001 laptop_mode
2153     0 sda1      26     0.003  3600.029  1290.730      0     0.000     0.000     0.000 rsyslogd
5575     0 sda1       0     0.000     0.000     0.000     16     0.000     0.000     0.000 cat
5581     0 sda1       0     0.000     0.000     0.000     12     0.001     0.002     0.000 perl
[output truncated]
Three applications stand out: 
 PID  UID  DEV WRITE_CNT WRITE_MIN WRITE_MAX WRITE_AVG READ_CNT READ_MIN READ_MAX READ_AVG COMMAND
2789  2903 sda1     854     0.000   120.000    39.836      0     0.000     0.000     0.000 plasma
2573     0 sda1      63     0.033  3600.015   515.226      0     0.000     0.000     0.000 auditd
2153     0 sda1      26     0.003  3600.029  1290.730      0     0.000     0.000     0.000 rsyslogd
These three applications have a WRITE_CNT greater than 0, which means that they performed some form of write during the measurement. Of those, plasma was the worst offender by a large degree: it performed the most write operations, and the average time between writes was the lowest. Plasma would therefore be the best candidate to investigate if you were concerned about power-inefficient applications.
Use the strace and ltrace commands to examine applications more closely by tracing all system calls of the given process ID. Run: 
strace -p 2789
The output of strace contains a repeating pattern every 45 seconds that opened the KDE icon cache file of the user for writing followed by an immediate close of the file again. This led to a necessary physical write to the hard disk as the file metadata (specifically, the modification time) had changed. The final fix was to prevent those unnecessary calls when no updates to the icons had occurred.
For reference on what the columns in the diskdevstat command stand for, see this table:

Table 2.1. Reading the diskdevstat Output

PIDthe process ID of the application
UIDthe user ID under which the applications is running
DEVthe device on which the I/O took place
WRITE_CNTthe total number of write operations
WRITE_MINthe lowest time taken for two consecutive writes (in seconds)
WRITE_MAXthe greatest time taken for two consecutive writes (in seconds)
WRITE_AVGthe average time taken for two consecutive writes (in seconds)
READ_CNTthe total number of read operations
READ_MINthe lowest time taken for two consecutive reads (in seconds)
READ_MAXthe greatest time taken for two consecutive reads (in seconds)
READ_AVGthe average time taken for two consecutive reads (in seconds)
COMMANDthe name of the process