CPUs were stuck in lock_kernel() function in Veritas environment

Issue

• System shows soft lockup in lock_kernel() function with the below message

BUG: soft lockup - CPU#1 stuck for 67s! [opcmona:14096]
Modules linked in: iptable_filter ip_tables mptctl mptbase vxodm(P)(U) vxgms(P)(U) amf(P)(U) vxglm(P)(U) vxfen(P)(U) gab(P)(U) llt(P)(U) rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr autofs4 nfs lockd fscache auth_rpcgss nfs_acl sunrpc dmpjbod(P)(U) dmpap(P)(U) dmpaa(P)(U) cpufreq_ondemand freq_table pcc_cpufreq bonding ipv6 vxcafs(P)(U) vxportal(P)(U) fdd(P)(U) vxfs(P)(U) exportfs dm_multipath vxspec(P)(U) vxio(P)(U) vxdmp(P)(U) ipmi_devintf microcode power_meter acpi_ipmi ipmi_si ipmi_msghandler iTCO_wdt iTCO_vendor_support hpilo hpwdt sg tg3(U) igb(U) i2c_algo_bit i2c_core ptp pps_core serio_raw lpc_ich mfd_core ioatdma dca shpchp ext4 jbd2 mbcache sd_mod lpfc(U) scsi_transport_fc scsi_tgt crc_t10dif hpsa(U) scsi_transport_sas pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
CPU 1 
Modules linked in: iptable_filter ip_tables mptctl mptbase vxodm(P)(U) vxgms(P)(U) amf(P)(U) vxglm(P)(U) vxfen(P)(U) gab(P)(U) llt(P)(U) rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr autofs4 nfs lockd fscache auth_rpcgss nfs_acl sunrpc dmpjbod(P)(U) dmpap(P)(U) dmpaa(P)(U) cpufreq_ondemand freq_table pcc_cpufreq bonding ipv6 vxcafs(P)(U) vxportal(P)(U) fdd(P)(U) vxfs(P)(U) exportfs dm_multipath vxspec(P)(U) vxio(P)(U) vxdmp(P)(U) ipmi_devintf microcode power_meter acpi_ipmi ipmi_si ipmi_msghandler iTCO_wdt iTCO_vendor_support hpilo hpwdt sg tg3(U) igb(U) i2c_algo_bit i2c_core ptp pps_core serio_raw lpc_ich mfd_core ioatdma dca shpchp ext4 jbd2 mbcache sd_mod lpfc(U) scsi_transport_fc scsi_tgt crc_t10dif hpsa(U) scsi_transport_sas pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 14096, comm: opcmona Tainted: P        W  -- ------------    2.6.32-754.6.3.el6.x86_64 #1 HP ProLiant DL380p Gen8
RIP: 0010:[<ffffffff8155c525>]  [<ffffffff8155c525>] lock_kernel+0x35/0x50
RSP: 0018:ffff885ff4d93da8  EFLAGS: 00000293
RAX: 0000000000000000 RBX: ffff885ff4d93da8 RCX: 0000000000000734
RDX: 0000000000000737 RSI: 00000000000000d0 RDI: 0000000000000246
RBP: ffffffff8156513e R08: ffff882ff2032608 R09: 0000000000000000
R10: 0000000000000053 R11: 0000000000000040 R12: 0000000100000000
R13: ffff885ff4d93e28 R14: ffffffff8124afd1 R15: ffff885ff4d93d78
FS:  00007f15790e0720(0000) GS:ffff880118440000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f64f5b65168 CR3: 0000002f8638e000 CR4: 00000000000607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process opcmona (pid: 14096, threadinfo ffff885ff4d90000, task ffff885ff5afaab0)
Stack:
 ffff885ff4d93e08 ffffffff811f4262 0000000000000000 0000000000000000
<d> ffff882fee76f088 0000000000000000 ffff882ff2032608 ffff882ff2032608
<d> ffff885ff4d93e78 ffff882ff2032648 ffff885ff5afaab0 ffff882ff2613080
Call Trace:
 [<ffffffff811f4262>] ? __posix_lock_file+0x62/0x500
 [<ffffffff811f4763>] ? vfs_lock_file+0x43/0x50
 [<ffffffff811f4997>] ? fcntl_setlk+0x167/0x330
 [<ffffffff811b6917>] ? sys_fcntl+0x197/0x530
 [<ffffffff810f2d8e>] ? __audit_syscall_exit+0x25e/0x290
 [<ffffffff81564357>] ? system_call_fastpath+0x35/0x3a
Code: 8b 40 1c 48 89 e5 83 c0 01 75 2d ba 00 00 01 00 f0 0f c1 15 6e a0 52 00 0f b7 ca c1 ea 10 39 d1 74 16 f3 90 0f b7 0d 5b a0 52 00 <eb> f1 83 3d 52 a0 52 00 00 75 ec eb d3 65 48 8b 14 25 00 fc 00 
Call Trace:
 [<ffffffff811f4262>] ? __posix_lock_file+0x62/0x500
 [<ffffffff811f4763>] ? vfs_lock_file+0x43/0x50
 [<ffffffff811f4997>] ? fcntl_setlk+0x167/0x330
 [<ffffffff811b6917>] ? sys_fcntl+0x197/0x530
 [<ffffffff810f2d8e>] ? __audit_syscall_exit+0x25e/0x290
 [<ffffffff81564357>] ? system_call_fastpath+0x35/0x3a

• Many other CPUs were also stuck in same lock_kernel().