Kernel panic due to task which was in a spin_lock call involving semaphores in RHEL 6.6.

Solution Unverified - Updated -

Issue

  • Kernel panic with below signature:
BUG: soft lockup - CPU#9 stuck for 67s! [dsmserv:13427]
Modules linked in: mptctl mptbase autofs4 lin_tape(U) nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand freq_table pcc_cpufreq ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 microcode iTCO_wdt iTCO_vendor_support hpilo tg3 ptp pps_core hpwdt serio_raw be2net lpc_ich mfd_core ioatdma dca sg ch osst st power_meter acpi_ipmi ipmi_si ipmi_msghandler shpchp ext4 jbd2 mbcache dm_round_robin sd_mod crc_t10dif sr_mod cdrom hpsa qla2xxx scsi_transport_fc scsi_tgt pata_acpi ata_generic ata_piix dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
CPU 9 
Modules linked in: mptctl mptbase autofs4 lin_tape(U) nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand freq_table pcc_cpufreq ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 microcode iTCO_wdt iTCO_vendor_support hpilo tg3 ptp pps_core hpwdt serio_raw be2net lpc_ich mfd_core ioatdma dca sg ch osst st power_meter acpi_ipmi ipmi_si ipmi_msghandler shpchp ext4 jbd2 mbcache dm_round_robin sd_mod crc_t10dif sr_mod cdrom hpsa qla2xxx scsi_transport_fc scsi_tgt pata_acpi ata_generic ata_piix dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 13427, comm: dsmserv Not tainted 2.6.32-504.el6.x86_64 #1 HP ProLiant ML350p Gen8
RIP: 0010:[<ffffffff8152ce6e>]  [<ffffffff8152ce6e>] _spin_lock+0x1e/0x30
RSP: 0018:ffff88082dd2bca8  EFLAGS: 00000297
RAX: 000000000000ffff RBX: ffff88082dd2bca8 RCX: 0000000000000000
RDX: 000000000000efed RSI: 0000000000000000 RDI: ffff88041eb3ffa0
RBP: ffffffff8100bb8e R08: ffff88082dd2a000 R09: 00000000ffffffff
R10: 00002bf260025695 R11: 0000000000000001 R12: ffffffff815299be
R13: ffff88082dd2bcd8 R14: ffff880435be7700 R15: 0000000000000009
FS:  00007ffff354e720(0000) GS:ffff88044e420000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000003516a03020 CR3: 000000082dcc9000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process dsmserv (pid: 13427, threadinfo ffff88082dd2a000, task ffff880733a58ae0)
Stack:
 ffff88082dd2bcd8 ffffffff8121c36c ffff88041eb3ff00 0000000000d98122
<d> 0000000000000000 0000000000000001 ffff88082dd2bf68 ffffffff8121e23e
<d> ffff88082dd2bd38 0000000000000000 8000000400000000 ffff880733a58ae0
Call Trace:
 [<ffffffff8121c36c>] ? sem_lock+0x6c/0x130
 [<ffffffff8121e23e>] ? sys_semtimedop+0x58e/0xaf0
 [<ffffffff8114fcea>] ? handle_mm_fault+0x22a/0x300
 [<ffffffff81063bf3>] ? perf_event_task_sched_out+0x33/0x70
 [<ffffffff8105e0d0>] ? __dequeue_entity+0x30/0x50
 [<ffffffff8100988e>] ? __switch_to+0x26e/0x320
 [<ffffffff815299be>] ? thread_return+0x4e/0x7d0
 [<ffffffff8118f981>] ? __fput+0x1a1/0x210
 [<ffffffff810e5c87>] ? audit_syscall_entry+0x1d7/0x200
 [<ffffffff8121e7b0>] ? sys_semop+0x10/0x20
 [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b

or



kernel: BUG: soft lockup - CPU#1 stuck for 67s! [db2fmp:43296] kernel: Modules linked in: autofs4 nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipv6 microcode ppdev vmware_balloon parport_pc parport sg i2c_piix4 i2c_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom vmxnet3 vmw_pvscsi pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib] kernel: CPU 1 kernel: Modules linked in: autofs4 nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipv6 microcode ppdev vmware_balloon parport_pc parport sg i2c_piix4 i2c_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom vmxnet3 vmw_pvscsi pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib] kernel: kernel: Pid: 43296, comm: db2fmp Not tainted 2.6.32-504.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform kernel: RIP: 0010:[<ffffffff8152ce6e>] [<ffffffff8152ce6e>] _spin_lock+0x1e/0x30 kernel: RSP: 0018:ffff8803808d5ca8 EFLAGS: 00000202 kernel: RAX: 000000000000ffff RBX: ffff8803808d5ca8 RCX: 0000000000000000 kernel: RDX: 000000000000efe9 RSI: 0000000000000000 RDI: ffff880102b13fa0 kernel: RBP: ffffffff8100bb8e R08: 0000000000000001 R09: 0000000000000000 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff815299be kernel: R13: ffff8803808d5cd8 R14: ffff8803b599cc40 R15: 0000000000000001 kernel: FS: 00007fffea1ac700(0000) GS:ffff880034a20000(0000) knlGS:0000000000000000 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 kernel: CR2: 00007ffff7ff0000 CR3: 00000003b6d81000 CR4: 00000000000007e0 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 kernel: Process db2fmp (pid: 43296, threadinfo ffff8803808d4000, task ffff8803b375caa0) kernel: Stack: kernel: ffff8803808d5cd8 ffffffff8121c36c ffff880102b13f00 0000000030870004 kernel: <d> 0000000000000000 0000000000000001 ffff8803808d5f68 ffffffff8121e23e kernel: <d> ffff8803808d5d38 0000000000000000 0000000000000000 ffff8803b375caa0 kernel: Call Trace: kernel: [<ffffffff8121c36c>] ? sem_lock+0x6c/0x130 kernel: [<ffffffff8121e23e>] ? sys_semtimedop+0x58e/0xaf0 kernel: [<ffffffff811df2e3>] ? locks_free_lock+0x43/0x60 kernel: [<ffffffff811df379>] ? locks_delete_lock+0x79/0xa0 kernel: [<ffffffff811dfe16>] ? __posix_lock_file+0x86/0x4e0 kernel: [<ffffffff8128c1c5>] ? _atomic_dec_and_lock+0x55/0x80 kernel: [<ffffffff811b07b0>] ? mntput_no_expire+0x30/0x110 kernel: [<ffffffff8118f981>] ? __fput+0x1a1/0x210 kernel: [<ffffffff8118fa15>] ? fput+0x25/0x30 kernel: [<ffffffff8118ac6d>] ? filp_close+0x5d/0x90 kernel: [<ffffffff8121e7b0>] ? sys_semop+0x10/0x20 kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b kernel: Code: 00 00 00 01 74 05 e8 22 bb d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89

Environment

  • Red Hat Enterprise Linux 6.6
  • kernel-2.6.32-504.el6
  • kernel-2.6.32-504.3.3.el6

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.