Frequent soft lockup of kblockd in Red Hat Enterprise Linux 5 with EMC PowerPath

Solution Verified - Updated -

Issue

Frequent hangs and soft lockups such as the following are seen:

Apr 20 09:51:29 zamdh20756 kernel: BUG: soft lockup - CPU#25 stuck for 10s! [kblockd/25:2250]
Apr 20 09:51:29 zamdh20756 kernel: CPU 25:
Apr 20 09:51:29 zamdh20756 kernel: Modules linked in: hangcheck_timer mptctl mptbase ipmi_si(U) ipmi_devintf(U) 
ipmi_msghandler(U) nfsd exportfs auth_rpcgss oracleasm(U) autofs4 
hp_ilo(U) hidp ocfs2(U) nfs lockd fscache nfs_acl rfcomm l2cap bluetooth ocfs2_dlmfs(U) ocfs2_
dlm(U) ocfs2_nodemanager(U) configfs sunrpc bonding emcpdm(PU) emcpgpx(PU) emcpmpx(PU) emcp(PU) dm_multipath video 
sbs backlight i2c_ec button battery asus_acpi acpi_memhotplug ac ipv6 

xfrm_nalgo crypto_api parport_pc lp parport sg i2c_piix4 e1000e qla2xxx i2c_core e1000(
U) ide_cd shpchp scsi_transport_fc pcspkr serio_raw cdrom dm_snapshot dm_zero dm_mirror dm_mod cciss sd_mod scsi_mod ext3 
jbd uhci_hcd ohci_hcd ehci_hcd
Apr 20 09:51:29 zamdh20756 kernel: Pid: 2250, comm: kblockd/25 Tainted: P      2.6.18-92.1.10.el5 #1
Apr 20 09:51:29 zamdh20756 kernel: RIP: 0010:[<ffffffff8005a63e>]  [<ffffffff8005a63e>] generic_unplug_device+0x27/0x28
Apr 20 09:51:29 zamdh20756 kernel: RSP: 0018:ffff81201feffe18  EFLAGS: 00000282
Apr 20 09:51:29 zamdh20756 kernel: RAX: ffff81181cf5a734 RBX: ffff81181d5227d8 RCX: ffff81181d5226b0
Apr 20 09:51:29 zamdh20756 kernel: RDX: ffff81083c2807c8 RSI: 0000000000000086 RDI: ffff8108170d3030
Apr 20 09:51:29 zamdh20756 kernel: RBP: 0000000000000086 R08: ffff81181d522790 R09: 0000000000000080
Apr 20 09:51:29 zamdh20756 kernel: R10: ffff81181fcdc338 R11: ffffffff8844d390 R12: ffff81181d5226b0
Apr 20 09:51:29 zamdh20756 kernel: R13: ffffffff8001c623 R14: 0000000000000086 R15: ffff81083c280000
Apr 20 09:51:29 zamdh20756 kernel: FS:  00002b48ef397c40(0000) GS:ffff81083c2a6240(0000) knlGS:00000000f7ee68d0
Apr 20 09:51:29 zamdh20756 kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Apr 20 09:51:29 zamdh20756 kernel: CR2: 000000048029e004 CR3: 00000036389b8000 CR4: 00000000000006e0
Apr 20 09:51:29 zamdh20756 kernel:
Apr 20 09:51:29 zamdh20756 kernel: Call Trace:
Apr 20 09:51:29 zamdh20756 kernel:  [<ffffffff8013ad2f>] blk_unplug_work+0x4b/0x50
Apr 20 09:51:29 zamdh20756 kernel:  [<ffffffff8004cf4c>] run_workqueue+0x94/0xe4
Apr 20 09:51:29 zamdh20756 kernel:  [<ffffffff8004982b>] worker_thread+0x0/0x122
Apr 20 09:51:29 zamdh20756 kernel:  [<ffffffff8009dd64>] keventd_create_kthread+0x0/0xc4
Apr 20 09:51:29 zamdh20756 kernel:  [<ffffffff8004991b>] worker_thread+0xf0/0x122
Apr 20 09:51:29 zamdh20756 kernel:  [<ffffffff8008ad7d>] default_wake_function+0x0/0xe
Apr 20 09:51:29 zamdh20756 kernel:  [<ffffffff8009dd64>] keventd_create_kthread+0x0/0xc4
Apr 20 09:51:29 zamdh20756 kernel:  [<ffffffff80032576>] kthread+0xfe/0x132
Apr 20 09:51:29 zamdh20756 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Apr 20 09:51:29 zamdh20756 kernel:  [<ffffffff8009dd64>] keventd_create_kthread+0x0/0xc4
Apr 20 09:51:29 zamdh20756 kernel:  [<ffffffff80032478>] kthread+0x0/0x132
Apr 20 09:51:29 zamdh20756 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11

Environment

  • Red Hat Enterprise Linux 5
  • EMC PowerPath 5.3 SP1 or earlier
  • Oracle RAC cluster
  • Many (>200) LUNs

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content