Chapter 44. Monitoring disk and I/O activity with SystemTap

You can monitor disk and I/O activity with the following scripts.

44.1. Summarizing disk read/write traffic with SystemTap

You can use the disktop.stp SystemTap script to identify which processes are performing the heaviest disk reads and writes to the system.

Prerequisites

Procedure

  • Run the disktop.stp script: 

    # stap --example disktop.stp

    The script displays the top ten processes responsible for the heaviest reads or writes to a disk.

    The output includes the following data per listed process:

    UID
    User ID. A user ID of 0 refers to the root user.
    PID
    The ID of the listed process.
    PPID
    The process ID of the listed process’s parent process.
    CMD
    The name of the listed process.
    DEVICE
    Which storage device the listed process is reading from or writing to.
    T
    The type of action performed by the listed process, where W refers to write, and R refers to read.
    BYTES
    The amount of data read to or written from disk.

Output of the disktop.stp script looks similar to the following: 

[...]
Mon Sep 29 03:38:28 2008 , Average:  19Kb/sec, Read: 7Kb, Write: 89Kb
UID      PID     PPID                       CMD   DEVICE    T    BYTES
0    26319    26294                   firefox     sda5    W        90229
0     2758     2757           pam_timestamp_c     sda5    R         8064
0     2885        1                     cupsd     sda5    W         1678
Mon Sep 29 03:38:38 2008 , Average:   1Kb/sec, Read: 7Kb, Write: 1Kb
UID      PID     PPID                       CMD   DEVICE    T    BYTES
0     2758     2757           pam_timestamp_c     sda5    R         8064
0     2885        1                     cupsd     sda5    W         1678

44.2. Tracking I/O time for each file read or write with SystemTap

You can use the iotime.stp SystemTap script to monitor the amount of time it takes for each process to read from or write to any file. This helps you to determine what files are slow to load on a system.

Prerequisites

Procedure

  • Run the iotime.stp script: 

    # stap --example iotime.stp

    The script tracks each time a system call opens, closes, reads from, and writes to a file. For each file any system call accesses, It counts the number of microseconds it takes for any reads or writes to finish and tracks the amount of data , in bytes, read from or written to the file.

    The output contains:

  • A timestamp, in microseconds
  • Process ID and process name
  • An access or iotime flag

  • The file accessed

    If a process was able to read or write any data, a pair of access and iotime lines should appear together. The access line refers to the time that a given process started accessing a file. The end of the access line will show the amount of data read or written. The iotime line will show the amount of time, in microseconds, that the process took in order to perform the read or write.

Output of the iotime.stp script looks similar to the following: 

[...]
825946 3364 (NetworkManager) access /sys/class/net/eth0/carrier read: 8190 write: 0
825955 3364 (NetworkManager) iotime /sys/class/net/eth0/carrier time: 9
[...]
117061 2460 (pcscd) access /dev/bus/usb/003/001 read: 43 write: 0
117065 2460 (pcscd) iotime /dev/bus/usb/003/001 time: 7
[...]
3973737 2886 (sendmail) access /proc/loadavg read: 4096 write: 0
3973744 2886 (sendmail) iotime /proc/loadavg time: 11
[...]

44.3. Tracking cumulative I/O with SystemTap

You can use the traceio.stp SystemTap script to track the cumulative amount of I/O to the system.

Prerequisites

Procedure

  • Run the traceio.stp script: 

    # stap --example traceio.stp

    The script prints the top ten executables generating I/O traffic over time. It also tracks the cumulative amount of I/O reads and writes done by those executables. This information is tracked and printed out in 1-second intervals, and in descending order.

    Output of the traceio.stp script looks similar to the following: 

[...]
           Xorg r:   583401 KiB w:        0 KiB
       floaters r:       96 KiB w:     7130 KiB
multiload-apple r:      538 KiB w:      537 KiB
           sshd r:       71 KiB w:       72 KiB
pam_timestamp_c r:      138 KiB w:        0 KiB
        staprun r:       51 KiB w:       51 KiB
          snmpd r:       46 KiB w:        0 KiB
          pcscd r:       28 KiB w:        0 KiB
     irqbalance r:       27 KiB w:        4 KiB
          cupsd r:        4 KiB w:       18 KiB
           Xorg r:   588140 KiB w:        0 KiB
       floaters r:       97 KiB w:     7143 KiB
multiload-apple r:      543 KiB w:      542 KiB
           sshd r:       72 KiB w:       72 KiB
pam_timestamp_c r:      138 KiB w:        0 KiB
        staprun r:       51 KiB w:       51 KiB
          snmpd r:       46 KiB w:        0 KiB
          pcscd r:       28 KiB w:        0 KiB
     irqbalance r:       27 KiB w:        4 KiB
          cupsd r:        4 KiB w:       18 KiB

44.4. Monitoring I/O activity on a specific device with SystemTap

You can use the traceio2.stp SystemTap script to monitor I/O activity on a specific device.

Prerequisites

Procedure

  • Run the traceio2.stp script. 
# stap --example traceio2.stp 'argument'

This script takes the whole device number as an argument. To find this number you can use: 

# stat -c "0x%D" directory

Where directory is located on the device you want to monitor.

The output contains following:

  • The name and ID of any process performing a read or write
  • The function it is performing (vfs_read or vfs_write)
  • The kernel device number

Consider following output of # stap traceio2.stp 0x805 

[...]
synergyc(3722) vfs_read 0x800005
synergyc(3722) vfs_read 0x800005
cupsd(2889) vfs_write 0x800005
cupsd(2889) vfs_write 0x800005
cupsd(2889) vfs_write 0x800005
[...]

44.5. Monitoring reads and writes to a file with SystemTap

You can use the inodewatch.stp SystemTap script to monitor reads from and writes to a file in real time.

Prerequisites

Procedure

  • Run the inodewatch.stp script. 
# stap --example inodewatch.stp 'argument1' 'argument2' 'argument3'

The script inodewatch.stp takes three command-line arguments:

  1. The file’s major device number.
  2. The file’s minor device number.
  3. The file’s inode number.

You can get these numbers using: 

# stat -c '%D %i' filename

Where filename is an absolute path.

Consider following example: 

# stat -c '%D %i' /etc/crontab

The output should look like: 

805 1078319

where:

  • 805 is the base-16 (hexadecimal) device number. The last two digits are the minor device number, and the remaining digits are the major number.
  • 1078319 is the inode number.

To start monitoring /etc/crontab, run: 

# stap inodewatch.stp 0x8 0x05 1078319

In the first two arguments you must use 0x prefixes for base-16 numbers.

The output contains following:

  • The name and ID of any process performing a read or write
  • The function it is performing (vfs_read or vfs_write)
  • The kernel device number

The output of this example should look like: 

cat(16437) vfs_read 0x800005/1078319
cat(16437) vfs_read 0x800005/1078319