Kernel panic due to task which was in a spin_lock call involving semaphores in RHEL 6.6.
Issue
- Kernel panic with below signature:
BUG: soft lockup - CPU#9 stuck for 67s! [dsmserv:13427]
Modules linked in: mptctl mptbase autofs4 lin_tape(U) nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand freq_table pcc_cpufreq ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 microcode iTCO_wdt iTCO_vendor_support hpilo tg3 ptp pps_core hpwdt serio_raw be2net lpc_ich mfd_core ioatdma dca sg ch osst st power_meter acpi_ipmi ipmi_si ipmi_msghandler shpchp ext4 jbd2 mbcache dm_round_robin sd_mod crc_t10dif sr_mod cdrom hpsa qla2xxx scsi_transport_fc scsi_tgt pata_acpi ata_generic ata_piix dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
CPU 9
Modules linked in: mptctl mptbase autofs4 lin_tape(U) nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand freq_table pcc_cpufreq ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 microcode iTCO_wdt iTCO_vendor_support hpilo tg3 ptp pps_core hpwdt serio_raw be2net lpc_ich mfd_core ioatdma dca sg ch osst st power_meter acpi_ipmi ipmi_si ipmi_msghandler shpchp ext4 jbd2 mbcache dm_round_robin sd_mod crc_t10dif sr_mod cdrom hpsa qla2xxx scsi_transport_fc scsi_tgt pata_acpi ata_generic ata_piix dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Pid: 13427, comm: dsmserv Not tainted 2.6.32-504.el6.x86_64 #1 HP ProLiant ML350p Gen8
RIP: 0010:[<ffffffff8152ce6e>] [<ffffffff8152ce6e>] _spin_lock+0x1e/0x30
RSP: 0018:ffff88082dd2bca8 EFLAGS: 00000297
RAX: 000000000000ffff RBX: ffff88082dd2bca8 RCX: 0000000000000000
RDX: 000000000000efed RSI: 0000000000000000 RDI: ffff88041eb3ffa0
RBP: ffffffff8100bb8e R08: ffff88082dd2a000 R09: 00000000ffffffff
R10: 00002bf260025695 R11: 0000000000000001 R12: ffffffff815299be
R13: ffff88082dd2bcd8 R14: ffff880435be7700 R15: 0000000000000009
FS: 00007ffff354e720(0000) GS:ffff88044e420000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000003516a03020 CR3: 000000082dcc9000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process dsmserv (pid: 13427, threadinfo ffff88082dd2a000, task ffff880733a58ae0)
Stack:
ffff88082dd2bcd8 ffffffff8121c36c ffff88041eb3ff00 0000000000d98122
<d> 0000000000000000 0000000000000001 ffff88082dd2bf68 ffffffff8121e23e
<d> ffff88082dd2bd38 0000000000000000 8000000400000000 ffff880733a58ae0
Call Trace:
[<ffffffff8121c36c>] ? sem_lock+0x6c/0x130
[<ffffffff8121e23e>] ? sys_semtimedop+0x58e/0xaf0
[<ffffffff8114fcea>] ? handle_mm_fault+0x22a/0x300
[<ffffffff81063bf3>] ? perf_event_task_sched_out+0x33/0x70
[<ffffffff8105e0d0>] ? __dequeue_entity+0x30/0x50
[<ffffffff8100988e>] ? __switch_to+0x26e/0x320
[<ffffffff815299be>] ? thread_return+0x4e/0x7d0
[<ffffffff8118f981>] ? __fput+0x1a1/0x210
[<ffffffff810e5c87>] ? audit_syscall_entry+0x1d7/0x200
[<ffffffff8121e7b0>] ? sys_semop+0x10/0x20
[<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
or
kernel: BUG: soft lockup - CPU#1 stuck for 67s! [db2fmp:43296]
kernel: Modules linked in: autofs4 nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipv6 microcode ppdev vmware_balloon parport_pc parport sg i2c_piix4 i2c_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom vmxnet3 vmw_pvscsi pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
kernel: CPU 1
kernel: Modules linked in: autofs4 nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipv6 microcode ppdev vmware_balloon parport_pc parport sg i2c_piix4 i2c_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom vmxnet3 vmw_pvscsi pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
kernel:
kernel: Pid: 43296, comm: db2fmp Not tainted 2.6.32-504.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
kernel: RIP: 0010:[<ffffffff8152ce6e>] [<ffffffff8152ce6e>] _spin_lock+0x1e/0x30
kernel: RSP: 0018:ffff8803808d5ca8 EFLAGS: 00000202
kernel: RAX: 000000000000ffff RBX: ffff8803808d5ca8 RCX: 0000000000000000
kernel: RDX: 000000000000efe9 RSI: 0000000000000000 RDI: ffff880102b13fa0
kernel: RBP: ffffffff8100bb8e R08: 0000000000000001 R09: 0000000000000000
kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff815299be
kernel: R13: ffff8803808d5cd8 R14: ffff8803b599cc40 R15: 0000000000000001
kernel: FS: 00007fffea1ac700(0000) GS:ffff880034a20000(0000) knlGS:0000000000000000
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 00007ffff7ff0000 CR3: 00000003b6d81000 CR4: 00000000000007e0
kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
kernel: Process db2fmp (pid: 43296, threadinfo ffff8803808d4000, task ffff8803b375caa0)
kernel: Stack:
kernel: ffff8803808d5cd8 ffffffff8121c36c ffff880102b13f00 0000000030870004
kernel: <d> 0000000000000000 0000000000000001 ffff8803808d5f68 ffffffff8121e23e
kernel: <d> ffff8803808d5d38 0000000000000000 0000000000000000 ffff8803b375caa0
kernel: Call Trace:
kernel: [<ffffffff8121c36c>] ? sem_lock+0x6c/0x130
kernel: [<ffffffff8121e23e>] ? sys_semtimedop+0x58e/0xaf0
kernel: [<ffffffff811df2e3>] ? locks_free_lock+0x43/0x60
kernel: [<ffffffff811df379>] ? locks_delete_lock+0x79/0xa0
kernel: [<ffffffff811dfe16>] ? __posix_lock_file+0x86/0x4e0
kernel: [<ffffffff8128c1c5>] ? _atomic_dec_and_lock+0x55/0x80
kernel: [<ffffffff811b07b0>] ? mntput_no_expire+0x30/0x110
kernel: [<ffffffff8118f981>] ? __fput+0x1a1/0x210
kernel: [<ffffffff8118fa15>] ? fput+0x25/0x30
kernel: [<ffffffff8118ac6d>] ? filp_close+0x5d/0x90
kernel: [<ffffffff8121e7b0>] ? sys_semop+0x10/0x20
kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
kernel: Code: 00 00 00 01 74 05 e8 22 bb d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89
Environment
- Red Hat Enterprise Linux 6.6
- kernel-2.6.32-504.el6
- kernel-2.6.32-504.3.3.el6
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
