After kernel upgrade md raid shows incorrect high %util values in iostat-x,sar and nmap output when the device is idle.

Issue

• iostat reports incorrect utilization for md raid device
• IO monitoring tools report md0 is at 100% utilization even when there is no, are hardly any, IO present on either the md or backing sd devices.
• monitoring tools report md0 has a high avgqu-sz along with 100% utilization when there is hardly any IO present on either the md or backing sd devices.
• sar -d and iostat -x report high disk utilization (%util)
  • note that the md devicesr/s and w/s are zero implying this device is 100% idle rather than 100% busy.
  • note that the sdN devices that back the md devices are all idle.

Device:   rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda         0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdb         0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdc         0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
md127       0.00     0.00    0.00    0.00     0.00     0.00     0.00    28.00    0.00    0.00    0.00   0.00 100.00
md126       0.00     0.00    0.00    0.00     0.00     0.00     0.00    28.00    0.00    0.00    0.00   0.00 100.00
md0         0.00     0.00    0.00    0.00     0.00     0.00     0.00    92.00    0.00    0.00    0.00   0.00 100.00

• sar -d and iostat -x may also report high avgqu-sz values in addition to high disk utilization when io is present on backing sdN disks.
  • note that the md devicesr/s and w/s are zero implying this device is 100% idle rather than 100% busy.
  • note that the sdN devices that back the md devices have a small amount of io present, but the avgqu-sz between sdN devices and md devices clearly do not match.

Device:   rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb         0.00     0.00    0.00    8.80     0.00     0.02     5.64     0.10   10.86    0.00   10.86  10.73   9.44
sda         0.00     0.00    0.00    8.60     0.00     0.02     5.77     0.09   10.02    0.00   10.02  10.56   9.08
md127       0.00     0.00    0.00    8.80     0.00     0.02     5.64 482426.54    0.00    0.00    0.00 113.64 100.00