System hangs for a short time during IO operations

Issue

• Virtual machine got hung during IO on disks
• System hangs for a short time during IO operations

• The vmstat output collected at the time of issue showed increase in blocked process count under procs->(b) column:

  $ vmstat 2
  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
   2 27      0 92563440 783672 27192124    0    0     1    36    1    2  0  0 93  6  0
   1 28      0 92562864 783676 27192124    0    0     0     4 6611 9633  7  1 14 79  0
   1 29      0 92563416 783684 27192124    0    0     0    12 6599 9591  7  1 12 81  0
   1 30      0 92564112 783684 27192128    0    0     0     0 6289 9349  7  1 12 81  0
   1 31      0 92564368 783684 27192128    0    0     0     8 6289 9419  7  0 12 81  0
   1 32      0 92564648 783684 27192128    0    0     0     0 6615 9861  7  1 12 80  0
   1 32      0 92564336 783684 27192128    0    0     0     0 6735 9921  7  1 12 80  0
   1 32      0 92564688 783684 27192128    0    0     0     8 6661 9865  7  1 12 81  0
   1 32      0 92564544 783684 27192584    0    0     0     6 6560 9592  7  1 12 81  0
   1 32      0 92564560 783684 27192584    0    0     0     0 6583 9609  6  1 12 81  0
   1 37      0 92564176 783688 27192584    0    0     0    26 6702 9792  7  1 11 82  0
   1 37      0 92564200 783688 27192584    0    0     0 16638 6628 9624  7  1  6 87  0
   1 37      0 92564720 783688 27192584    0    0     0     0 6419 9518  7  0  6 87  0
   1 38      0 92565104 783688 27192584    0    0     0    96 6596 9687  7  1  6 87  0
   1 38      0 92565232 783688 27192528    0    0     0     0 6559 9560  7  1  6 87  0
   1 38      0 92571648 783688 27192332    0    0     0     6 6705 10510  7  1  6 86  0
   1 38      0 92571856 783688 27192316    0    0     0     0 6371 9533  7  1  6 86  0
   1 38      0 92571984 783688 27192316    0    0     0     0 6072 8997  7  1  6 87  0
   2 38      0 92571328 783688 27192316    0    0     0     0 6348 9337  7  1  6 87  0
   4 38      0 92570768 783688 27192316    0    0     0     0 6237 9263  7  0  6 87  0
   1 38      0 92571112 783688 27192316    0    0     0     0 6072 9111  7  0  6 87  0
   2 38      0 92570576 783688 27192316    0    0     0     0 6508 9430  7  1  6 87  0
   1 40      0 92571280 783688 27192316    0    0     0     0 6484 9395  7  1  6 87  0
   6  0      0 92525280 783876 27191032    0    0     0   738 20738 32043  9  6 25 60  0
  

• The ps output showed number of jbd2 tasks and other applications, processes in D state:

  $ while true; do     ps aux | awk '$8 ~ /^D/{print}';     sleep 1; done
  root      7473  0.0  0.0      0     0 ?        D    Jan06   0:15 [jbd2/dm-6-8]
  root      8614  0.0  0.0      0     0 ?        D    Jan06   0:00 [jbd2/dm-34-8]
  patrol   51797  1.2  0.0 211844 34200 ?        DNs  Jan07 157:36 PatrolAgent -p 3181 -id eapaaaa
  root      7436  0.0  0.0      0     0 ?        D    Jan06   0:09 [jbd2/dm-3-8]
  root      7470  0.0  0.0      0     0 ?        D    Jan06   0:16 [jbd2/dm-10-8]
  root      7473  0.0  0.0      0     0 ?        D    Jan06   0:15 [jbd2/dm-6-8]
  root      8614  0.0  0.0      0     0 ?        D    Jan06   0:00 [jbd2/dm-34-8]
  root      8686  0.0  0.0      0     0 ?        D    Jan06   0:10 [jbd2/dm-45-8]
  root     19812  0.0  0.0  62044  1064 ?        D<sl 10:02   0:00 /sbin/auditd
  root     23424  0.0  0.0      0     0 ?        D    06:43   0:00 [kworker/4:2]
  root     19812  0.0  0.0  62044  1064 ?        D<sl 10:02   0:00 /sbin/auditd