Frequent soft lockup of kblockd in Red Hat Enterprise Linux 5 with EMC PowerPath
Issue
Frequent hangs and soft lockups such as the following are seen:
Apr 20 09:51:29 zamdh20756 kernel: BUG: soft lockup - CPU#25 stuck for 10s! [kblockd/25:2250]
Apr 20 09:51:29 zamdh20756 kernel: CPU 25:
Apr 20 09:51:29 zamdh20756 kernel: Modules linked in: hangcheck_timer mptctl mptbase ipmi_si(U) ipmi_devintf(U)
ipmi_msghandler(U) nfsd exportfs auth_rpcgss oracleasm(U) autofs4
hp_ilo(U) hidp ocfs2(U) nfs lockd fscache nfs_acl rfcomm l2cap bluetooth ocfs2_dlmfs(U) ocfs2_
dlm(U) ocfs2_nodemanager(U) configfs sunrpc bonding emcpdm(PU) emcpgpx(PU) emcpmpx(PU) emcp(PU) dm_multipath video
sbs backlight i2c_ec button battery asus_acpi acpi_memhotplug ac ipv6
xfrm_nalgo crypto_api parport_pc lp parport sg i2c_piix4 e1000e qla2xxx i2c_core e1000(
U) ide_cd shpchp scsi_transport_fc pcspkr serio_raw cdrom dm_snapshot dm_zero dm_mirror dm_mod cciss sd_mod scsi_mod ext3
jbd uhci_hcd ohci_hcd ehci_hcd
Apr 20 09:51:29 zamdh20756 kernel: Pid: 2250, comm: kblockd/25 Tainted: P 2.6.18-92.1.10.el5 #1
Apr 20 09:51:29 zamdh20756 kernel: RIP: 0010:[<ffffffff8005a63e>] [<ffffffff8005a63e>] generic_unplug_device+0x27/0x28
Apr 20 09:51:29 zamdh20756 kernel: RSP: 0018:ffff81201feffe18 EFLAGS: 00000282
Apr 20 09:51:29 zamdh20756 kernel: RAX: ffff81181cf5a734 RBX: ffff81181d5227d8 RCX: ffff81181d5226b0
Apr 20 09:51:29 zamdh20756 kernel: RDX: ffff81083c2807c8 RSI: 0000000000000086 RDI: ffff8108170d3030
Apr 20 09:51:29 zamdh20756 kernel: RBP: 0000000000000086 R08: ffff81181d522790 R09: 0000000000000080
Apr 20 09:51:29 zamdh20756 kernel: R10: ffff81181fcdc338 R11: ffffffff8844d390 R12: ffff81181d5226b0
Apr 20 09:51:29 zamdh20756 kernel: R13: ffffffff8001c623 R14: 0000000000000086 R15: ffff81083c280000
Apr 20 09:51:29 zamdh20756 kernel: FS: 00002b48ef397c40(0000) GS:ffff81083c2a6240(0000) knlGS:00000000f7ee68d0
Apr 20 09:51:29 zamdh20756 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Apr 20 09:51:29 zamdh20756 kernel: CR2: 000000048029e004 CR3: 00000036389b8000 CR4: 00000000000006e0
Apr 20 09:51:29 zamdh20756 kernel:
Apr 20 09:51:29 zamdh20756 kernel: Call Trace:
Apr 20 09:51:29 zamdh20756 kernel: [<ffffffff8013ad2f>] blk_unplug_work+0x4b/0x50
Apr 20 09:51:29 zamdh20756 kernel: [<ffffffff8004cf4c>] run_workqueue+0x94/0xe4
Apr 20 09:51:29 zamdh20756 kernel: [<ffffffff8004982b>] worker_thread+0x0/0x122
Apr 20 09:51:29 zamdh20756 kernel: [<ffffffff8009dd64>] keventd_create_kthread+0x0/0xc4
Apr 20 09:51:29 zamdh20756 kernel: [<ffffffff8004991b>] worker_thread+0xf0/0x122
Apr 20 09:51:29 zamdh20756 kernel: [<ffffffff8008ad7d>] default_wake_function+0x0/0xe
Apr 20 09:51:29 zamdh20756 kernel: [<ffffffff8009dd64>] keventd_create_kthread+0x0/0xc4
Apr 20 09:51:29 zamdh20756 kernel: [<ffffffff80032576>] kthread+0xfe/0x132
Apr 20 09:51:29 zamdh20756 kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11
Apr 20 09:51:29 zamdh20756 kernel: [<ffffffff8009dd64>] keventd_create_kthread+0x0/0xc4
Apr 20 09:51:29 zamdh20756 kernel: [<ffffffff80032478>] kthread+0x0/0x132
Apr 20 09:51:29 zamdh20756 kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11
Environment
- Red Hat Enterprise Linux 5
- EMC PowerPath 5.3 SP1 or earlier
- Oracle RAC cluster
- Many (>200) LUNs
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.