unmounted disks still has io going to the disk per /proc/diskstats and iostat, why?

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux 4.6

Issue

  • /proc/diskstats shows outstanding io to unmounted device

       8   32 sdc 562126 445 67809663 1918357 4902068 0 39216544 4214925992 451 1583514511 2995158897
                  1      2   3        4       5       6 7        8          9   10         11

    • Field #9 above is the number of io currently in-progress to the device, so 451 io are outstanding to this device in this case.
  • iostat shows read io completing against the disk

Resolution

There are several likely causes, the most common one is a file system signature scanner.

The queued commands could be a result of lvm attempting to  scan the device for metadata.  From the sosreport, we see the lvm filter is  currently configured to scan this and other devices:

/etc/lvm/lvm.conf:

    # By default we accept every block device:

    filter = [ "a/sda[1-9]$/" "a/sdb[1-9]$/" "a/sdc[1-9]$/" "a/sdd[1-9]$/" "a/emcpower.*/" "r/.*/" ]

From the pvs output within sosreport, it appears none of the /dev/sdN devices are a PV, so they can be safely removed from the scan list via the above filter.

  PV              VG        Fmt  Attr PSize   PFree DevSize PV UUID                              
  /dev/emcpowera1               --        0      0   20.00G
  /dev/emcpowerb  prod5vg  lvm2 a-     5.00G    0    5.00G dpWmjp-ByB9-vwHK-oHxb-X0XR-2dQ5-jEVcUK
  /dev/emcpowerc  prod6vg  lvm2 a-     5.00G    0    5.00G HU6UrH-SyRH-2vDm-6Chl-Ucoj-yPMF-KkYI5m
  /dev/emcpowerd  prod3vg  lvm2 a-   268.00G    0  268.00G pgzPN6-QeSB-EgbY-rwBn-mkO6-Ylhc-pIzP7D
  /dev/emcpowere  orabinvg lvm2 a-   100.00G    0  100.00G Qw2C1G-Lnvq-vR5P-hOAK-x3zD-bRTx-qKE3R6
  /dev/emcpowerf  homevg    lvm2 a-   50.00G    0   50.00G cp5gHm-R1wp-ebQ9-q0I7-2J2k-tFLS-jhfPcm
  /dev/emcpowerg  archvg   lvm2 a-   200.00G    0  200.00G gkcrxe-U1FI-K2aH-ZMfn-O9g0-Lnz2-9tBigx
  /dev/emcpowerh  prod1vg  lvm2 a-   400.00G    0  400.00G yhz2Ee-ZveX-zbsr-v2FO-YoL0-3Ng4-5BtRDA
  /dev/emcpoweri  prod2vg  lvm2 a-   400.00G    0  400.00G foVt7i-c4Qp-F06K-H751-y32f-kdaV-JSUlc7
  /dev/emcpowerj  prod4vg  lvm2 a-     5.00G    0    5.00G PoA275-Z4GS-c8wj-pDxf-vO6J-ssYN-jDji5X
  /dev/sda1                      --        0     0  101.94M 
  /dev/sda2                      --        0     0    64.22G
  /dev/sda3                      --        0     0    1.95G
  /dev/sdb1                      --        0     0   18.64G
  /dev/sdb5                      --        0     0  117.98G
  /dev/sdc1                      --        0     0  409.86G

The lvm filter can be changed to prevent lvm from issuing io to the device(s) in question:

/etc/lvm/lvm.conf:

    # By default we accept every block device:

    filter = [ "a/emcpower.*/" "r/.*/" ]

This will allow all emcpower devices, but reject (exclude) all other block devices including sda, sdb, and sdc in this case.

Once the above change to lvm.conf has been made, several steps need to be taken to make the change active.

1. Clear lvm cache:
# rm -f /etc/lvm/cache/.cache

2. Regenerate cache file:
# pvscan
# lvscan
# vgscan

Then verify the above commands result in correct operation, that is pvscan should output /dev/emcpowerX devices etc. and iostat and/or /proc/diskstats should no io being sent to the devices.

3. Finally, rebuild the initrd image file to include this new lvm.conf, for example:
# cp /boot/initrd-2.6.9-67.0.7.ELhugemem.img /boot/initrd-2.6.9-67.0.7.ELhugemem.img.BAK
# mkinitrd -v -f /boot/initrd-2.6.9-67.0.7.ELhugemem.img 2.6.9-67.0.7.ELhugemem

Root Cause

LVM issues io to allowed devices scanning for on-disk PV information.

Diagnostic Steps

If lvm is not the cause or sole cause of the io to the device, then a system tap script may be required to help identify the program or programs that are issuing io to the unmounted device.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments