System crashed with "Watchdog detected hard LOCKUP on cpu XX" error.

Issue

• System crashed with following panic message

Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 12
Pid: 14813, comm: sacf Tainted: GF          ---------------  T 2.6.32-279.el6.x86_64 #1
Call Trace:
 <NMI> ..
..
 [<ffffffff815008b0>] ? nmi+0x20/0x30
 [<ffffffff8150000f>] ? _spin_lock_irqsave+0x2f/0x40
 <<EOE>>  [<ffffffff81096481>] ? lock_hrtimer_base+0x31/0x60
 [<ffffffff810972c7>] ? hrtimer_try_to_cancel+0x27/0xd0
 [<ffffffff810a3b49>] ? futex_wait_queue_me+0xb9/0xf0
 [<ffffffff81097392>] ? hrtimer_cancel+0x22/0x30
 [<ffffffff810a4c47>] ? futex_wait+0x227/0x380
 [<ffffffff81096230>] ? hrtimer_wakeup+0x0/0x30
 [<ffffffff81097294>] ? hrtimer_start_range_ns+0x14/0x20
 [<ffffffff810a6361>] ? do_futex+0x121/0xb00
 [<ffffffff810edb54>] ? rb_reserve_next_event+0xb4/0x370
 [<ffffffff810edfc2>] ? ring_buffer_lock_reserve+0xa2/0x160
 [<ffffffff810d69e2>] ? audit_syscall_entry+0x272/0x2a0
 [<ffffffff810a6dbb>] ? sys_futex+0x7b/0x170
 [<ffffffff8100b308>] ? tracesys+0xd9/0xde

Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 3
Pid: 0, comm: swapper Not tainted 2.6.32-431.el6.x86_64 #1
Call Trace:
 <NMI> ..
..
 [<ffffffff8152aaa0>] ? nmi+0x20/0x30
 [<ffffffff812889c8>] ? __rb_rotate_left+0x8/0x90
 <<EOE>>  <IRQ>  [<ffffffff81288b7d>] ? rb_insert_color+0x9d/0x160
 [<ffffffff8109f342>] ? enqueue_hrtimer+0x82/0xd0
 [<ffffffff810a0117>] ? __hrtimer_start_range_ns+0x177/0x470
 [<ffffffff810a0428>] ? hrtimer_start+0x18/0x20
 [<ffffffff8147d8bf>] ? qdisc_watchdog_schedule+0x2f/0x40
..
 [<ffffffff8107a8e1>] ? __do_softirq+0xc1/0x1e0
 [<ffffffff810e6ec0>] ? handle_IRQ_event+0x60/0x170
 [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
 [<ffffffff8100fa75>] ? do_softirq+0x65/0xa0
 [<ffffffff8107a795>] ? irq_exit+0x85/0x90
 [<ffffffff81530fe5>] ? do_IRQ+0x75/0xf0
 [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
 <EOI> ..
..

#12 [ffff880028267f50] nmi+0x20 at ffffffff8152aaa0
    [exception RIP: __rb_rotate_left+0x8]                          <<< this may differ
    RIP: ffffffff812889c8  RSP: ffff8800282637d0  RFLAGS: 00000086
    RAX: ffff882064d4ebc0  RBX: ffff882064d4ebc0  RCX: ffff882064d4ebc0
    RDX: ffff882064d4ebc0  RSI: ffff880028271638  RDI: ffff882064d4ebc0
    RBP: ffff880028263810   R8: 0000000000000003   R9: 0000000000000001
    R10: 0000000000000001  R11: 0000000000000000  R12: ffff882064d4ebc0
    R13: ffff882064d4ebc0  R14: ffff882064d4ebc0  R15: ffff882064d4ebc0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
--- <NMI exception stack> ---
#13 [ffff8800282637d0] __rb_rotate_left+0x8 at ffffffff812889c8    <<< this may differ or miss
#14 [ffff8800282637d8] rb_insert_color+0x9d at ffffffff81288b7d    <<< code was looping here
#15 [ffff880028263818] enqueue_hrtimer+0x82 at ffffffff8109f342
#16 [ffff880028263848] __hrtimer_start_range_ns+0x177 at ffffffff810a0117
#17 [ffff8800282638e8] hrtimer_start+0x18 at ffffffff810a0428
#18 [ffff8800282638f8] qdisc_watchdog_schedule+0x2f at ffffffff8147d8bf
..
#37 [ffff880028263ec8] __do_softirq+0xc1 at ffffffff8107a8e1
#38 [ffff880028263f38] call_softirq+0x1c at ffffffff8100c30c
#39 [ffff880028263f50] do_softirq+0x65 at ffffffff8100fa75
#40 [ffff880028263f70] irq_exit+0x85 at ffffffff8107a795
#41 [ffff880028263f80] do_IRQ+0x75 at ffffffff81530fe5
--- <IRQ stack> ---