RHEL5 or RHEL6: kernel crash, kblockd crashes at _spin_lock_irqsave due to invalid spinlock_t structure, called from PowerGetSemaNew

Solution In Progress - Updated -

Issue

  • RHEL5 or RHEL6 kernel crash with kblockd crashing inside _spin_lock_irqsave called from PowerGetSemaNew.
  • Example of RHEL5 kernel 2.6.18-308.el5 crash
PID: 1127   TASK: ffff81203fb79860  CPU: 9   COMMAND: "kblockd/9"
 #0 [ffff81203f6e1b60] crash_kexec at ffffffff800b0938
 #1 [ffff81203f6e1c20] __die at ffffffff80065137
 #2 [ffff81203f6e1c60] die at ffffffff8006c789
 #3 [ffff81203f6e1c90] do_general_protection at ffffffff8006557f
 #4 [ffff81203f6e1cd0] error_exit at ffffffff8005dde9
    [exception RIP: _spin_lock_irqsave+3]
    RIP: ffffffff80064aeb  RSP: ffff81203f6e1d88  RFLAGS: 00010086
    RAX: 0000000000000286  RBX: 3032342c30303236  RCX: ffffffff8029f6f3
    RDX: 0000000000000000  RSI: 0000000000000001  RDI: 3032342c30303236
    RBP: 0000000000000001   R8: ffff810e7241e1e0   R9: ffff81203fb79860
    R10: ffff81184f99b300  R11: 0000000000000060  R12: ffff81201e96a008
    R13: ffff81201e96a030  R14: ffff81202890e0c8  R15: ffff81201e96a008
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0000
 #5 [ffff81203f6e1d88] PowerGetSemaNew at ffffffff885068e2 [emcp]
 #6 [ffff81203f6e1db8] PowerTopDispatch at ffffffff88508e70 [emcp]
 #7 [ffff81203f6e1de8] emcp_pseudo_strategy_request_fn at ffffffff8850d0ab [emcp]
 #8 [ffff81203f6e1e28] generic_unplug_device at ffffffff8005a464
 #9 [ffff81203f6e1e38] run_workqueue at ffffffff8004d2e4
#10 [ffff81203f6e1e78] worker_thread at ffffffff80049bdd
#11 [ffff81203f6e1ee8] kthread at ffffffff80032679
#12 [ffff81203f6e1f48] kernel_thread at ffffffff8005dfb1
  • Example of RHEL6 kernel 2.6.32-358.el6 crash
PID: 154    TASK: ffff880c2cdb8ae0  CPU: 0   COMMAND: "kblockd/0"
 #0 [ffff880c2ce9f9d0] machine_kexec at ffffffff81035b7b
 #1 [ffff880c2ce9fa30] crash_kexec at ffffffff810c0db2
 #2 [ffff880c2ce9fb00] oops_end at ffffffff815111d0
 #3 [ffff880c2ce9fb30] die at ffffffff8100f19b
 #4 [ffff880c2ce9fb60] do_general_protection at ffffffff81510cd2
 #5 [ffff880c2ce9fb90] general_protection at ffffffff815104a5
    [exception RIP: _spin_lock_irqsave+31]
    RIP: ffffffff8150feaf  RSP: ffff880c2ce9fc40  RFLAGS: 00010086
    RAX: 0000000000010000  RBX: f570235003f12a2b  RCX: 00000000f6495faa
    RDX: 0000000000000286  RSI: 0000000000000001  RDI: f570235003f12a2b
    RBP: ffff880c2ce9fc40   R8: 0000000000000800   R9: 0000000000001000
    R10: 0000000000001000  R11: 0000000000000000  R12: 0000000000000001
    R13: ffff88182aaacf28  R14: ffff8801d2c40d00  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0000
 #6 [ffff880c2ce9fc48] PowerGetSemaNew at ffffffffa04910bb [emcp]
 #7 [ffff880c2ce9fc78] PowerGetSema at ffffffffa04912d3 [emcp]
 #8 [ffff880c2ce9fc88] PowerPlatformBottomDispatch at ffffffffa0497b35 [emcp]
 #9 [ffff880c2ce9fce8] PowerBottomDispatch at ffffffffa0498088 [emcp]
#10 [ffff880c2ce9fd18] PowerBottomDispatchPirp at ffffffffa04983a9 [emcp]
#11 [ffff880c2ce9fd48] PowerTopDispatch at ffffffffa049dda0 [emcp]
#12 [ffff880c2ce9fd78] emcp_common_strategy_request_fn at ffffffffa049e116 [emcp]
#13 [ffff880c2ce9fdd8] emcp_pseudo_strategy_request_fn at ffffffffa049e3be [emcp]
#14 [ffff880c2ce9fdf8] __blk_run_queue at ffffffff8125b381
#15 [ffff880c2ce9fe18] cfq_kick_queue at ffffffff81270979
#16 [ffff880c2ce9fe38] worker_thread at ffffffff81090ac0
#17 [ffff880c2ce9fee8] kthread at ffffffff81096916
#18 [ffff880c2ce9ff48] kernel_thread at ffffffff8100c0ca

Environment

  • Red Hat Enterprise Linux 5
    • seen on kernel-2.6.18-308.el5 with EMCpower.LINUX-5.6.0.00.00-143.x86_64
  • Red Hat Enterprise Linux 6
    • seen on kernel-2.6.32-279.el6
    • seen on kernel-2.6.32-358.el6 with EMC PowerPath 5.7 SP3 (EMCpower.LINUX-5.7.3.00.00-029.el6.x86_64)
  • EMC PowerPath kernel modules
    • emcp

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.