RHEL5 or RHEL6: kernel crash, kblockd crashes at _spin_lock_irqsave due to invalid spinlock_t structure, called from PowerGetSemaNew
Issue
- RHEL5 or RHEL6 kernel crash with kblockd crashing inside _spin_lock_irqsave called from PowerGetSemaNew.
- Example of RHEL5 kernel 2.6.18-308.el5 crash
PID: 1127 TASK: ffff81203fb79860 CPU: 9 COMMAND: "kblockd/9"
#0 [ffff81203f6e1b60] crash_kexec at ffffffff800b0938
#1 [ffff81203f6e1c20] __die at ffffffff80065137
#2 [ffff81203f6e1c60] die at ffffffff8006c789
#3 [ffff81203f6e1c90] do_general_protection at ffffffff8006557f
#4 [ffff81203f6e1cd0] error_exit at ffffffff8005dde9
[exception RIP: _spin_lock_irqsave+3]
RIP: ffffffff80064aeb RSP: ffff81203f6e1d88 RFLAGS: 00010086
RAX: 0000000000000286 RBX: 3032342c30303236 RCX: ffffffff8029f6f3
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 3032342c30303236
RBP: 0000000000000001 R8: ffff810e7241e1e0 R9: ffff81203fb79860
R10: ffff81184f99b300 R11: 0000000000000060 R12: ffff81201e96a008
R13: ffff81201e96a030 R14: ffff81202890e0c8 R15: ffff81201e96a008
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000
#5 [ffff81203f6e1d88] PowerGetSemaNew at ffffffff885068e2 [emcp]
#6 [ffff81203f6e1db8] PowerTopDispatch at ffffffff88508e70 [emcp]
#7 [ffff81203f6e1de8] emcp_pseudo_strategy_request_fn at ffffffff8850d0ab [emcp]
#8 [ffff81203f6e1e28] generic_unplug_device at ffffffff8005a464
#9 [ffff81203f6e1e38] run_workqueue at ffffffff8004d2e4
#10 [ffff81203f6e1e78] worker_thread at ffffffff80049bdd
#11 [ffff81203f6e1ee8] kthread at ffffffff80032679
#12 [ffff81203f6e1f48] kernel_thread at ffffffff8005dfb1
- Example of RHEL6 kernel 2.6.32-358.el6 crash
PID: 154 TASK: ffff880c2cdb8ae0 CPU: 0 COMMAND: "kblockd/0"
#0 [ffff880c2ce9f9d0] machine_kexec at ffffffff81035b7b
#1 [ffff880c2ce9fa30] crash_kexec at ffffffff810c0db2
#2 [ffff880c2ce9fb00] oops_end at ffffffff815111d0
#3 [ffff880c2ce9fb30] die at ffffffff8100f19b
#4 [ffff880c2ce9fb60] do_general_protection at ffffffff81510cd2
#5 [ffff880c2ce9fb90] general_protection at ffffffff815104a5
[exception RIP: _spin_lock_irqsave+31]
RIP: ffffffff8150feaf RSP: ffff880c2ce9fc40 RFLAGS: 00010086
RAX: 0000000000010000 RBX: f570235003f12a2b RCX: 00000000f6495faa
RDX: 0000000000000286 RSI: 0000000000000001 RDI: f570235003f12a2b
RBP: ffff880c2ce9fc40 R8: 0000000000000800 R9: 0000000000001000
R10: 0000000000001000 R11: 0000000000000000 R12: 0000000000000001
R13: ffff88182aaacf28 R14: ffff8801d2c40d00 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000
#6 [ffff880c2ce9fc48] PowerGetSemaNew at ffffffffa04910bb [emcp]
#7 [ffff880c2ce9fc78] PowerGetSema at ffffffffa04912d3 [emcp]
#8 [ffff880c2ce9fc88] PowerPlatformBottomDispatch at ffffffffa0497b35 [emcp]
#9 [ffff880c2ce9fce8] PowerBottomDispatch at ffffffffa0498088 [emcp]
#10 [ffff880c2ce9fd18] PowerBottomDispatchPirp at ffffffffa04983a9 [emcp]
#11 [ffff880c2ce9fd48] PowerTopDispatch at ffffffffa049dda0 [emcp]
#12 [ffff880c2ce9fd78] emcp_common_strategy_request_fn at ffffffffa049e116 [emcp]
#13 [ffff880c2ce9fdd8] emcp_pseudo_strategy_request_fn at ffffffffa049e3be [emcp]
#14 [ffff880c2ce9fdf8] __blk_run_queue at ffffffff8125b381
#15 [ffff880c2ce9fe18] cfq_kick_queue at ffffffff81270979
#16 [ffff880c2ce9fe38] worker_thread at ffffffff81090ac0
#17 [ffff880c2ce9fee8] kthread at ffffffff81096916
#18 [ffff880c2ce9ff48] kernel_thread at ffffffff8100c0ca
Environment
- Red Hat Enterprise Linux 5
- seen on kernel-2.6.18-308.el5 with EMCpower.LINUX-5.6.0.00.00-143.x86_64
- Red Hat Enterprise Linux 6
- seen on kernel-2.6.32-279.el6
- seen on kernel-2.6.32-358.el6 with EMC PowerPath 5.7 SP3 (EMCpower.LINUX-5.7.3.00.00-029.el6.x86_64)
- EMC PowerPath kernel modules
- emcp
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.