System was crashed by the NMI watchdog, CPU was stuck in the spin_lock_irqsave() called by cfq_exit_io_context().

Issue

• One of the servers crashed, RCA required.
• System crashed with following panic message

NMI Watchdog detected LOCKUP on CPU 1
CPU 1 
Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 nfs lockd fscache nfs_acl sunrpc bonding ip_conntrack_netbios_
ns ipt_LOG xt_tcpudp xt_state ip_conntrack nfnetlink iptable_filter ip_tables x_tables dm_round_robin dm_multipath v
ideo sbs backlight i2c_ec i2c_core button battery asus_acpi acpi_memhotplug ac parport_pc lp parport shpchp bnx2 tg3
 i5000_edac edac_mc pcspkr dm_snapshot dm_zero dm_mirror dm_mod mppVhba(U) usb_storage ata_piix libata mptsas mptscs
ih scsi_transport_sas mptbase qla2xxx scsi_transport_fc mppUpper(U) sg sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd eh
ci_hcd
Pid: 15989, comm: scsi_id Tainted: G      2.6.18-92.1.18.el5 #1
RIP: 0010:[<ffffffff80064bb0>]  [<ffffffff80064bb0>] .text.lock.spinlock+0xe/0x30
RSP: 0018:ffff810a7d2bbed0  EFLAGS: 00000082
RAX: 0000000000000296 RBX: ffff810179a6e870 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffffffff80306e60
RBP: ffff810179a6e840 R08: ffff810f2e3b6c90 R09: ffff81013806bc80
R10: ffff810a7d2bbc68 R11: 0000000000000000 R12: 0000000000000000
R13: ffff81026fb198d8 R14: 0000000000427780 R15: 0000000000427740
FS:  0000000000000000(0000) GS:ffff8101380f77c0(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000000049b9c0 CR3: 0000000000201000 CR4: 00000000000006e0
Process scsi_id (pid: 15989, threadinfo ffff810a7d2ba000, task ffff81026fb197a0)
Stack:  ffffffff8003258e 0000000000000038 0000000000000292 ffff810179a6e840
 0000000000000000 ffff81026fb198d8 ffffffff8005a8ac 0000000000000000
 0000000000000010 ffff81026fb197a0 ffffffff8001580a 0000000000000000
Call Trace:
 [<ffffffff8003258e>] cfq_exit_io_context+0x1d/0x13f
 [<ffffffff8005a8ac>] exit_io_context+0x8c/0x97
 [<ffffffff8001580a>] do_exit+0x8b5/0x8f1
 [<ffffffff80048a84>] cpuset_exit+0x0/0x6c
 [<ffffffff8005d28d>] tracesys+0xd5/0xe0