System experiencing high load after a kernel upgrade in a Red Hat Enterprise Linux 5 system
Issue
-
The following tasks are seen to be stuck in D state :
[...] $ grep " D" ps root 16 0.0 0.0 0 0 ? D< 11:39 0:00 [events/2] root 4522 0.0 0.0 10784 372 ? Ds 11:40 0:00 irqbalance root 4742 0.0 0.0 158612 8632 ? Dl 11:40 0:00 /usr/sbin/snmpd -Lsd -Lf /dev/null -p /var/run/snmpd.pid -a root 19934 0.0 0.0 2184 804 ? D 11:52 0:00 dup_inv -print root 20139 0.0 0.0 3808 416 ? Ds 11:52 0:00 /sbin/mingetty tty2 root 20140 0.0 0.0 3808 412 ? Ds 11:52 0:00 /sbin/mingetty tty3 root 20141 0.0 0.0 3808 412 ? Ds 11:52 0:00 /sbin/mingetty tty4 root 20142 0.0 0.0 3808 416 ? Ds 11:52 0:00 /sbin/mingetty tty5 root 20145 0.0 0.0 3808 416 ? Ds 11:52 0:00 /sbin/mingetty tty6 [...] -
This causes the load average to shootup :
$ cat uptime 12:23:32 up 44 min, 2 users, load average: 13.16, 12.97, 10.99 -
The following traces are seen in the logs :
dup_inv D ffff81103ff33100 0 7821 6621 (NOTLB) ffff810fae78da78 0000000000000086 0000000010008040 0000000000000007 ffff81103df5e080 0000000000000007 ffff81103df5e080 ffff81103ff33100 000000e4590695be 0000000000000ada ffff81103df5e268 000000003d3e4080 Call Trace: [<ffffffff8008e20d>] __activate_task+0x56/0x6d [<ffffffff80063c53>] __mutex_lock_slowpath+0x60/0x9b [<ffffffff80063c9d>] .text.lock.mutex+0xf/0x14 [<ffffffff8009ff6f>] destroy_workqueue+0x1c/0x8a [<ffffffff880760c0>] :scsi_mod:scsi_host_dev_release+0x54/0x96 [<ffffffff80154ef0>] kobject_cleanup+0x53/0x7e [<ffffffff80154f1b>] kobject_release+0x0/0x9 [<ffffffff80035336>] kref_put+0x5d/0x68 [<ffffffff884f107b>] :cnic:cnic_ulp_start+0x6b/0x87 [<ffffffff884f81ca>] :cnic:cnic_netdev_event+0x529/0x610 [<ffffffff8002dfd7>] __wake_up+0x38/0x4f [<ffffffff800569bd>] netlink_broadcast+0x2a9/0x2f1 [<ffffffff80269552>] inetdev_event+0x20/0x282 [<ffffffff800675bb>] notifier_call_chain+0x20/0x32 [<ffffffff8023760a>] dev_open+0x96/0x9c [<ffffffff802356ff>] dev_change_flags+0x5a/0x119 [<ffffffff80269efe>] devinet_ioctl+0x235/0x59c [<ffffffff8022c816>] sock_ioctl+0x1c1/0x1e5 [<ffffffff80041ef3>] do_ioctl+0x21/0x6b [<ffffffff8002fff5>] vfs_ioctl+0x457/0x4b9 [<ffffffff8004c299>] sys_ioctl+0x59/0x78 [<ffffffff8010028d>] dev_ifsioc+0xdd/0x202 [<ffffffff800fe2b9>] compat_sys_ioctl+0x26d/0x2b1 [<ffffffff8006149d>] sysenter_do_call+0x1e/0x76 mingetty D ffff810001016120 0 8009 1 8010 8008 (NOTLB) ffff810fac0eddb8 0000000000000086 0000000000000096 0000000000000003 ffff810fac0edd48 0000000000000007 ffff810fae059100 ffff810138f22080 000000e45683dfcf 000000000004543f ffff810fae0592e8 000000023a719818 Call Trace: [<ffffffff8003a673>] tty_ldisc_deref+0x6a/0x7d [<ffffffff8009fe4a>] flush_cpu_workqueue+0x83/0xba [<ffffffff800a2dfd>] autoremove_wake_function+0x0/0x2e [<ffffffff80063ae9>] mutex_lock+0xd/0x1d [<ffffffff8009fee1>] flush_workqueue+0x60/0x87 [<ffffffff801ae37c>] release_dev+0x503/0x67b [<ffffffff800534a1>] tty_release+0x11/0x1a [<ffffffff80012c22>] __fput+0xd3/0x1bd [<ffffffff80023daf>] filp_close+0x5c/0x64 [<ffffffff8001e211>] sys_close+0x88/0xbd [<ffffffff8005d28d>] tracesys+0xd5/0xe0 mingetty D ffff81000101e7a0 0 8010 1 8011 8009 (NOTLB) ffff810fac063da8 0000000000000086 0000000000000000 0000000000000001 0000000000000096 0000000000000006 ffff810fad4e4820 ffff810138f96100 000000e456ac9429 000000000000daac ffff810fad4e4a08 00000003b0f1cc80 Call Trace: [<ffffffff80063c53>] __mutex_lock_slowpath+0x60/0x9b [<ffffffff8003d85b>] lock_timer_base+0x1b/0x3c [<ffffffff80063c9d>] .text.lock.mutex+0xf/0x14 [<ffffffff8009fec0>] flush_workqueue+0x3f/0x87 [<ffffffff801ae37c>] release_dev+0x503/0x67b [<ffffffff800490cd>] chrdev_open+0x0/0x183 [<ffffffff800534a1>] tty_release+0x11/0x1a [<ffffffff80012c22>] __fput+0xd3/0x1bd [<ffffffff80023daf>] filp_close+0x5c/0x64 [<ffffffff8001e211>] sys_close+0x88/0xbd [<ffffffff8005d28d>] tracesys+0xd5/0xe0
Environment
- Red Hat Enterprise Linux 5.7
- Broadcom NetXtreme II CNIC Driver
- Kernel 2.6.18-274
- Kernel 2.6.18-274.3.1
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
