System experiencing high load after a kernel upgrade in a Red Hat Enterprise Linux 5 system

Solution Verified - Updated -

Issue

  • The following tasks are seen to be stuck in D state :

    [...]
    $ grep " D" ps
    root        16  0.0  0.0      0     0 ?        D<   11:39   0:00 [events/2]
    root      4522  0.0  0.0  10784   372 ?        Ds   11:40   0:00 irqbalance
    root      4742  0.0  0.0 158612  8632 ?        Dl   11:40   0:00 /usr/sbin/snmpd -Lsd -Lf /dev/null -p /var/run/snmpd.pid -a
    root     19934  0.0  0.0   2184   804 ?        D    11:52   0:00 dup_inv -print
    root     20139  0.0  0.0   3808   416 ?        Ds   11:52   0:00 /sbin/mingetty tty2
    root     20140  0.0  0.0   3808   412 ?        Ds   11:52   0:00 /sbin/mingetty tty3
    root     20141  0.0  0.0   3808   412 ?        Ds   11:52   0:00 /sbin/mingetty tty4
    root     20142  0.0  0.0   3808   416 ?        Ds   11:52   0:00 /sbin/mingetty tty5
    root     20145  0.0  0.0   3808   416 ?        Ds   11:52   0:00 /sbin/mingetty tty6
    [...]
    
  • This causes the load average to shootup :

    $ cat uptime 
     12:23:32 up 44 min,  2 users,  load average: 13.16, 12.97, 10.99
    
  • The following traces are seen in the logs :

    dup_inv       D ffff81103ff33100     0  7821   6621                     (NOTLB)
     ffff810fae78da78 0000000000000086 0000000010008040 0000000000000007
     ffff81103df5e080 0000000000000007 ffff81103df5e080 ffff81103ff33100
     000000e4590695be 0000000000000ada ffff81103df5e268 000000003d3e4080
    Call Trace:
     [<ffffffff8008e20d>] __activate_task+0x56/0x6d
     [<ffffffff80063c53>] __mutex_lock_slowpath+0x60/0x9b
     [<ffffffff80063c9d>] .text.lock.mutex+0xf/0x14
     [<ffffffff8009ff6f>] destroy_workqueue+0x1c/0x8a
     [<ffffffff880760c0>] :scsi_mod:scsi_host_dev_release+0x54/0x96
     [<ffffffff80154ef0>] kobject_cleanup+0x53/0x7e
     [<ffffffff80154f1b>] kobject_release+0x0/0x9
     [<ffffffff80035336>] kref_put+0x5d/0x68
     [<ffffffff884f107b>] :cnic:cnic_ulp_start+0x6b/0x87
     [<ffffffff884f81ca>] :cnic:cnic_netdev_event+0x529/0x610
     [<ffffffff8002dfd7>] __wake_up+0x38/0x4f
     [<ffffffff800569bd>] netlink_broadcast+0x2a9/0x2f1
     [<ffffffff80269552>] inetdev_event+0x20/0x282
     [<ffffffff800675bb>] notifier_call_chain+0x20/0x32
     [<ffffffff8023760a>] dev_open+0x96/0x9c
     [<ffffffff802356ff>] dev_change_flags+0x5a/0x119
     [<ffffffff80269efe>] devinet_ioctl+0x235/0x59c
     [<ffffffff8022c816>] sock_ioctl+0x1c1/0x1e5
     [<ffffffff80041ef3>] do_ioctl+0x21/0x6b
     [<ffffffff8002fff5>] vfs_ioctl+0x457/0x4b9
     [<ffffffff8004c299>] sys_ioctl+0x59/0x78
     [<ffffffff8010028d>] dev_ifsioc+0xdd/0x202
     [<ffffffff800fe2b9>] compat_sys_ioctl+0x26d/0x2b1
     [<ffffffff8006149d>] sysenter_do_call+0x1e/0x76
    
    
    
    
    mingetty      D ffff810001016120     0  8009      1          8010  8008 (NOTLB)
     ffff810fac0eddb8 0000000000000086 0000000000000096 0000000000000003
     ffff810fac0edd48 0000000000000007 ffff810fae059100 ffff810138f22080
     000000e45683dfcf 000000000004543f ffff810fae0592e8 000000023a719818
    Call Trace:
     [<ffffffff8003a673>] tty_ldisc_deref+0x6a/0x7d
     [<ffffffff8009fe4a>] flush_cpu_workqueue+0x83/0xba
     [<ffffffff800a2dfd>] autoremove_wake_function+0x0/0x2e
     [<ffffffff80063ae9>] mutex_lock+0xd/0x1d
     [<ffffffff8009fee1>] flush_workqueue+0x60/0x87
     [<ffffffff801ae37c>] release_dev+0x503/0x67b
     [<ffffffff800534a1>] tty_release+0x11/0x1a
     [<ffffffff80012c22>] __fput+0xd3/0x1bd
     [<ffffffff80023daf>] filp_close+0x5c/0x64
     [<ffffffff8001e211>] sys_close+0x88/0xbd
     [<ffffffff8005d28d>] tracesys+0xd5/0xe0
    
    mingetty      D ffff81000101e7a0     0  8010      1          8011  8009 (NOTLB)
     ffff810fac063da8 0000000000000086 0000000000000000 0000000000000001
     0000000000000096 0000000000000006 ffff810fad4e4820 ffff810138f96100
     000000e456ac9429 000000000000daac ffff810fad4e4a08 00000003b0f1cc80
    Call Trace:
     [<ffffffff80063c53>] __mutex_lock_slowpath+0x60/0x9b
     [<ffffffff8003d85b>] lock_timer_base+0x1b/0x3c
     [<ffffffff80063c9d>] .text.lock.mutex+0xf/0x14
     [<ffffffff8009fec0>] flush_workqueue+0x3f/0x87
     [<ffffffff801ae37c>] release_dev+0x503/0x67b
     [<ffffffff800490cd>] chrdev_open+0x0/0x183
     [<ffffffff800534a1>] tty_release+0x11/0x1a
     [<ffffffff80012c22>] __fput+0xd3/0x1bd
     [<ffffffff80023daf>] filp_close+0x5c/0x64
     [<ffffffff8001e211>] sys_close+0x88/0xbd
     [<ffffffff8005d28d>] tracesys+0xd5/0xe0
    

Environment

  • Red Hat Enterprise Linux 5.7
  • Broadcom NetXtreme II CNIC Driver
  • Kernel 2.6.18-274
  • Kernel 2.6.18-274.3.1

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.