System experiencing high load after a kernel upgrade in a Red Hat Enterprise Linux 5 system

Issue

• The following tasks are seen to be stuck in D state :

  [...]
  $ grep " D" ps
  root        16  0.0  0.0      0     0 ?        D<   11:39   0:00 [events/2]
  root      4522  0.0  0.0  10784   372 ?        Ds   11:40   0:00 irqbalance
  root      4742  0.0  0.0 158612  8632 ?        Dl   11:40   0:00 /usr/sbin/snmpd -Lsd -Lf /dev/null -p /var/run/snmpd.pid -a
  root     19934  0.0  0.0   2184   804 ?        D    11:52   0:00 dup_inv -print
  root     20139  0.0  0.0   3808   416 ?        Ds   11:52   0:00 /sbin/mingetty tty2
  root     20140  0.0  0.0   3808   412 ?        Ds   11:52   0:00 /sbin/mingetty tty3
  root     20141  0.0  0.0   3808   412 ?        Ds   11:52   0:00 /sbin/mingetty tty4
  root     20142  0.0  0.0   3808   416 ?        Ds   11:52   0:00 /sbin/mingetty tty5
  root     20145  0.0  0.0   3808   416 ?        Ds   11:52   0:00 /sbin/mingetty tty6
  [...]
  

• This causes the load average to shootup :

  $ cat uptime 
   12:23:32 up 44 min,  2 users,  load average: 13.16, 12.97, 10.99
  

• The following traces are seen in the logs :

  dup_inv       D ffff81103ff33100     0  7821   6621                     (NOTLB)
   ffff810fae78da78 0000000000000086 0000000010008040 0000000000000007
   ffff81103df5e080 0000000000000007 ffff81103df5e080 ffff81103ff33100
   000000e4590695be 0000000000000ada ffff81103df5e268 000000003d3e4080
  Call Trace:
   [<ffffffff8008e20d>] __activate_task+0x56/0x6d
   [<ffffffff80063c53>] __mutex_lock_slowpath+0x60/0x9b
   [<ffffffff80063c9d>] .text.lock.mutex+0xf/0x14
   [<ffffffff8009ff6f>] destroy_workqueue+0x1c/0x8a
   [<ffffffff880760c0>] :scsi_mod:scsi_host_dev_release+0x54/0x96
   [<ffffffff80154ef0>] kobject_cleanup+0x53/0x7e
   [<ffffffff80154f1b>] kobject_release+0x0/0x9
   [<ffffffff80035336>] kref_put+0x5d/0x68
   [<ffffffff884f107b>] :cnic:cnic_ulp_start+0x6b/0x87
   [<ffffffff884f81ca>] :cnic:cnic_netdev_event+0x529/0x610
   [<ffffffff8002dfd7>] __wake_up+0x38/0x4f
   [<ffffffff800569bd>] netlink_broadcast+0x2a9/0x2f1
   [<ffffffff80269552>] inetdev_event+0x20/0x282
   [<ffffffff800675bb>] notifier_call_chain+0x20/0x32
   [<ffffffff8023760a>] dev_open+0x96/0x9c
   [<ffffffff802356ff>] dev_change_flags+0x5a/0x119
   [<ffffffff80269efe>] devinet_ioctl+0x235/0x59c
   [<ffffffff8022c816>] sock_ioctl+0x1c1/0x1e5
   [<ffffffff80041ef3>] do_ioctl+0x21/0x6b
   [<ffffffff8002fff5>] vfs_ioctl+0x457/0x4b9
   [<ffffffff8004c299>] sys_ioctl+0x59/0x78
   [<ffffffff8010028d>] dev_ifsioc+0xdd/0x202
   [<ffffffff800fe2b9>] compat_sys_ioctl+0x26d/0x2b1
   [<ffffffff8006149d>] sysenter_do_call+0x1e/0x76
  
  
  
  
  mingetty      D ffff810001016120     0  8009      1          8010  8008 (NOTLB)
   ffff810fac0eddb8 0000000000000086 0000000000000096 0000000000000003
   ffff810fac0edd48 0000000000000007 ffff810fae059100 ffff810138f22080
   000000e45683dfcf 000000000004543f ffff810fae0592e8 000000023a719818
  Call Trace:
   [<ffffffff8003a673>] tty_ldisc_deref+0x6a/0x7d
   [<ffffffff8009fe4a>] flush_cpu_workqueue+0x83/0xba
   [<ffffffff800a2dfd>] autoremove_wake_function+0x0/0x2e
   [<ffffffff80063ae9>] mutex_lock+0xd/0x1d
   [<ffffffff8009fee1>] flush_workqueue+0x60/0x87
   [<ffffffff801ae37c>] release_dev+0x503/0x67b
   [<ffffffff800534a1>] tty_release+0x11/0x1a
   [<ffffffff80012c22>] __fput+0xd3/0x1bd
   [<ffffffff80023daf>] filp_close+0x5c/0x64
   [<ffffffff8001e211>] sys_close+0x88/0xbd
   [<ffffffff8005d28d>] tracesys+0xd5/0xe0
  
  mingetty      D ffff81000101e7a0     0  8010      1          8011  8009 (NOTLB)
   ffff810fac063da8 0000000000000086 0000000000000000 0000000000000001
   0000000000000096 0000000000000006 ffff810fad4e4820 ffff810138f96100
   000000e456ac9429 000000000000daac ffff810fad4e4a08 00000003b0f1cc80
  Call Trace:
   [<ffffffff80063c53>] __mutex_lock_slowpath+0x60/0x9b
   [<ffffffff8003d85b>] lock_timer_base+0x1b/0x3c
   [<ffffffff80063c9d>] .text.lock.mutex+0xf/0x14
   [<ffffffff8009fec0>] flush_workqueue+0x3f/0x87
   [<ffffffff801ae37c>] release_dev+0x503/0x67b
   [<ffffffff800490cd>] chrdev_open+0x0/0x183
   [<ffffffff800534a1>] tty_release+0x11/0x1a
   [<ffffffff80012c22>] __fput+0xd3/0x1bd
   [<ffffffff80023daf>] filp_close+0x5c/0x64
   [<ffffffff8001e211>] sys_close+0x88/0xbd
   [<ffffffff8005d28d>] tracesys+0xd5/0xe0