Kernel panic due to task which was in a spin_lock call involving semaphores in RHEL 6.6.

Issue

• Kernel panic with below signature:

BUG: soft lockup - CPU#9 stuck for 67s! [dsmserv:13427]
Modules linked in: mptctl mptbase autofs4 lin_tape(U) nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand freq_table pcc_cpufreq ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 microcode iTCO_wdt iTCO_vendor_support hpilo tg3 ptp pps_core hpwdt serio_raw be2net lpc_ich mfd_core ioatdma dca sg ch osst st power_meter acpi_ipmi ipmi_si ipmi_msghandler shpchp ext4 jbd2 mbcache dm_round_robin sd_mod crc_t10dif sr_mod cdrom hpsa qla2xxx scsi_transport_fc scsi_tgt pata_acpi ata_generic ata_piix dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
CPU 9 
Modules linked in: mptctl mptbase autofs4 lin_tape(U) nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand freq_table pcc_cpufreq ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 microcode iTCO_wdt iTCO_vendor_support hpilo tg3 ptp pps_core hpwdt serio_raw be2net lpc_ich mfd_core ioatdma dca sg ch osst st power_meter acpi_ipmi ipmi_si ipmi_msghandler shpchp ext4 jbd2 mbcache dm_round_robin sd_mod crc_t10dif sr_mod cdrom hpsa qla2xxx scsi_transport_fc scsi_tgt pata_acpi ata_generic ata_piix dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 13427, comm: dsmserv Not tainted 2.6.32-504.el6.x86_64 #1 HP ProLiant ML350p Gen8
RIP: 0010:[<ffffffff8152ce6e>]  [<ffffffff8152ce6e>] _spin_lock+0x1e/0x30
RSP: 0018:ffff88082dd2bca8  EFLAGS: 00000297
RAX: 000000000000ffff RBX: ffff88082dd2bca8 RCX: 0000000000000000
RDX: 000000000000efed RSI: 0000000000000000 RDI: ffff88041eb3ffa0
RBP: ffffffff8100bb8e R08: ffff88082dd2a000 R09: 00000000ffffffff
R10: 00002bf260025695 R11: 0000000000000001 R12: ffffffff815299be
R13: ffff88082dd2bcd8 R14: ffff880435be7700 R15: 0000000000000009
FS:  00007ffff354e720(0000) GS:ffff88044e420000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000003516a03020 CR3: 000000082dcc9000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process dsmserv (pid: 13427, threadinfo ffff88082dd2a000, task ffff880733a58ae0)
Stack:
 ffff88082dd2bcd8 ffffffff8121c36c ffff88041eb3ff00 0000000000d98122
<d> 0000000000000000 0000000000000001 ffff88082dd2bf68 ffffffff8121e23e
<d> ffff88082dd2bd38 0000000000000000 8000000400000000 ffff880733a58ae0
Call Trace:
 [<ffffffff8121c36c>] ? sem_lock+0x6c/0x130
 [<ffffffff8121e23e>] ? sys_semtimedop+0x58e/0xaf0
 [<ffffffff8114fcea>] ? handle_mm_fault+0x22a/0x300
 [<ffffffff81063bf3>] ? perf_event_task_sched_out+0x33/0x70
 [<ffffffff8105e0d0>] ? __dequeue_entity+0x30/0x50
 [<ffffffff8100988e>] ? __switch_to+0x26e/0x320
 [<ffffffff815299be>] ? thread_return+0x4e/0x7d0
 [<ffffffff8118f981>] ? __fput+0x1a1/0x210
 [<ffffffff810e5c87>] ? audit_syscall_entry+0x1d7/0x200
 [<ffffffff8121e7b0>] ? sys_semop+0x10/0x20
 [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b

or

kernel: BUG: soft lockup - CPU#1 stuck for 67s! [db2fmp:43296]
kernel: Modules linked in: autofs4 nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipv6 microcode ppdev vmware_balloon parport_pc parport sg i2c_piix4 i2c_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom vmxnet3 vmw_pvscsi pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
kernel: CPU 1 
kernel: Modules linked in: autofs4 nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipv6 microcode ppdev vmware_balloon parport_pc parport sg i2c_piix4 i2c_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom vmxnet3 vmw_pvscsi pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
kernel: 
kernel: Pid: 43296, comm: db2fmp Not tainted 2.6.32-504.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
kernel: RIP: 0010:[<ffffffff8152ce6e>]  [<ffffffff8152ce6e>] _spin_lock+0x1e/0x30
kernel: RSP: 0018:ffff8803808d5ca8  EFLAGS: 00000202
kernel: RAX: 000000000000ffff RBX: ffff8803808d5ca8 RCX: 0000000000000000
kernel: RDX: 000000000000efe9 RSI: 0000000000000000 RDI: ffff880102b13fa0
kernel: RBP: ffffffff8100bb8e R08: 0000000000000001 R09: 0000000000000000
kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff815299be
kernel: R13: ffff8803808d5cd8 R14: ffff8803b599cc40 R15: 0000000000000001
kernel: FS:  00007fffea1ac700(0000) GS:ffff880034a20000(0000) knlGS:0000000000000000
kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 00007ffff7ff0000 CR3: 00000003b6d81000 CR4: 00000000000007e0
kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
kernel: Process db2fmp (pid: 43296, threadinfo ffff8803808d4000, task ffff8803b375caa0)
kernel: Stack:
kernel: ffff8803808d5cd8 ffffffff8121c36c ffff880102b13f00 0000000030870004
kernel: <d> 0000000000000000 0000000000000001 ffff8803808d5f68 ffffffff8121e23e
kernel: <d> ffff8803808d5d38 0000000000000000 0000000000000000 ffff8803b375caa0
kernel: Call Trace:
kernel: [<ffffffff8121c36c>] ? sem_lock+0x6c/0x130
kernel: [<ffffffff8121e23e>] ? sys_semtimedop+0x58e/0xaf0
kernel: [<ffffffff811df2e3>] ? locks_free_lock+0x43/0x60
kernel: [<ffffffff811df379>] ? locks_delete_lock+0x79/0xa0
kernel: [<ffffffff811dfe16>] ? __posix_lock_file+0x86/0x4e0
kernel: [<ffffffff8128c1c5>] ? _atomic_dec_and_lock+0x55/0x80
kernel: [<ffffffff811b07b0>] ? mntput_no_expire+0x30/0x110
kernel: [<ffffffff8118f981>] ? __fput+0x1a1/0x210
kernel: [<ffffffff8118fa15>] ? fput+0x25/0x30
kernel: [<ffffffff8118ac6d>] ? filp_close+0x5d/0x90
kernel: [<ffffffff8121e7b0>] ? sys_semop+0x10/0x20
kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
kernel: Code: 00 00 00 01 74 05 e8 22 bb d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89