Watchdog detected hard LOCKUP on cpu "X" in __smp_call_function_single()

Issue

• Watchdog detected hard LOCKUP on cpu "X" in __smp_call_function_single() and kernel panic with following call traces.

Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0
Pid: 23281, comm: qemu-kvm Tainted: P   M       ----------------   2.6.32-131.0.15.el6.x86_64 #1
Call Trace:
 <NMI>  [<ffffffff814dac28>] ? panic+0x78/0x143
 [<ffffffff810d649d>] ? watchdog_overflow_callback+0xcd/0xd0
 [<ffffffff81108b56>] ? __perf_event_overflow+0x116/0x290
 [<ffffffff81109149>] ? perf_event_overflow+0x19/0x20
 [<ffffffff8101dd56>] ? intel_pmu_handle_irq+0x316/0x550
 [<ffffffff814e06c6>] ? kprobe_exceptions_notify+0x16/0x430
 [<ffffffff814df1b8>] ? perf_event_nmi_handler+0x58/0xe0
 [<ffffffff814e0cf5>] ? notifier_call_chain+0x55/0x80
 [<ffffffff814e0d5a>] ? atomic_notifier_call_chain+0x1a/0x20
 [<ffffffff810940fe>] ? notify_die+0x2e/0x30
 [<ffffffff814de963>] ? do_nmi+0x173/0x2b0
 [<ffffffff814de270>] ? nmi+0x20/0x30
 [<ffffffff810a416e>] ? __smp_call_function_single+0x4e/0xe0
 <<EOE>>  <IRQ>  [<ffffffff810571c5>] ? scheduler_tick+0x225/0x280
 [<ffffffff8109dff0>] ? tick_sched_timer+0x0/0xc0
 [<ffffffff81079902>] ? update_process_times+0x52/0x70
 [<ffffffff8109e056>] ? tick_sched_timer+0x66/0xc0
 [<ffffffff8109289e>] ? __run_hrtimer+0x8e/0x1a0
 [<ffffffff81012979>] ? read_tsc+0x9/0x20
 [<ffffffff81092c46>] ? hrtimer_interrupt+0xe6/0x250
 [<ffffffff814e35eb>] ? smp_apic_timer_interrupt+0x6b/0x9b
 [<ffffffff8100bc93>] ? apic_timer_interrupt+0x13/0x20
 [<ffffffff814ddb27>] ? _spin_unlock_irqrestore+0x17/0x20
 [<ffffffff8104f843>] ? __wake_up+0x53/0x70
 [<ffffffffa01e1495>] ? tun_net_xmit+0x1a5/0x200 [tun]
 [<ffffffff8141e758>] ? dev_hard_start_xmit+0x2c8/0x3f0
 [<ffffffff81439d0a>] ? sch_direct_xmit+0x15a/0x1c0
 [<ffffffff81423098>] ? dev_queue_xmit+0x388/0x4d0
 [<ffffffffa0325400>] ? br_dev_queue_push_xmit+0x0/0xa0 [bridge]
 [<ffffffffa0325500>] ? __br_forward+0x0/0xc0 [bridge]
 [<ffffffffa032546c>] ? br_dev_queue_push_xmit+0x6c/0xa0 [bridge]
 [<ffffffffa03254f8>] ? br_forward_finish+0x58/0x60 [bridge]
 [<ffffffffa032559a>] ? __br_forward+0x9a/0xc0 [bridge]
 [<ffffffff81413993>] ? skb_clone+0x43/0x70
 [<ffffffffa032513e>] ? deliver_clone+0x3e/0x60 [bridge]
 [<ffffffffa032535c>] ? br_flood+0x7c/0xe0 [bridge]
 [<ffffffffa0325500>] ? __br_forward+0x0/0xc0 [bridge]
 [<ffffffffa03253d5>] ? br_flood_forward+0x15/0x20 [bridge]
 [<ffffffffa032647e>] ? br_handle_frame_finish+0x23e/0x260 [bridge]
 [<ffffffffa032664a>] ? br_handle_frame+0x1aa/0x250 [bridge]
 [<ffffffff8141dba3>] ? __netif_receive_skb+0x1d3/0x6b0
 [<ffffffff81420048>] ? netif_receive_skb+0x58/0x60
 [<ffffffff81420150>] ? napi_skb_finish+0x50/0x70
 [<ffffffff81422489>] ? napi_gro_receive+0x39/0x50
 [<ffffffffa0106bfe>] ? ixgbe_clean_rx_irq+0x51e/0x8a0 [ixgbe]
 [<ffffffffa010739f>] ? ixgbe_clean_rxtx_many+0x10f/0x220 [ixgbe]
 [<ffffffff814225a3>] ? net_rx_action+0x103/0x2f0
 [<ffffffff8106f717>] ? __do_softirq+0xb7/0x1e0
 [<ffffffff810d6940>] ? handle_IRQ_event+0x60/0x170
 [<ffffffff8106f776>] ? __do_softirq+0x116/0x1e0
 [<ffffffff8100c2cc>] ? call_softirq+0x1c/0x30
 [<ffffffff8100df05>] ? do_softirq+0x65/0xa0
 [<ffffffff8106f505>] ? irq_exit+0x85/0x90
 [<ffffffff814e3505>] ? do_IRQ+0x75/0xf0
 [<ffffffff8100bad3>] ? ret_from_intr+0x0/0x11
 <EOI>  [<ffffffff814ddb27>] ? _spin_unlock_irqrestore+0x17/0x20
 [<ffffffff810a40d1>] ? generic_exec_single+0x71/0xc0
 [<ffffffffa01cf640>] ? __vcpu_clear+0x0/0x90 [kvm_intel]
 [<ffffffff810a4352>] ? smp_call_function_single+0x152/0x160
 [<ffffffffa01cf640>] ? __vcpu_clear+0x0/0x90 [kvm_intel]
 [<ffffffffa01cf7cd>] ? vmx_vcpu_load+0x4d/0x130 [kvm_intel]
 [<ffffffffa01cd106>] ? __vmx_load_host_state+0xe6/0x100 [kvm_intel]
 [<ffffffffa017ae29>] ? kvm_arch_vcpu_load+0x29/0x100 [kvm]
 [<ffffffffa0170bbf>] ? vcpu_load+0x3f/0x50 [kvm]
 [<ffffffffa0170c4d>] ? kvm_vcpu_block+0x7d/0xc0 [kvm]
 [<ffffffff8108e160>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa0183ed9>] ? kvm_arch_vcpu_ioctl_run+0x469/0xec0 [kvm]
 [<ffffffffa016e332>] ? kvm_vcpu_ioctl+0x522/0x670 [kvm]
 [<ffffffff8105dc72>] ? default_wake_function+0x12/0x20
 [<ffffffff811876f6>] ? pollwake+0x56/0x60
 [<ffffffff8105dc60>] ? default_wake_function+0x0/0x20
 [<ffffffff8104af29>] ? __wake_up_common+0x59/0x90
 [<ffffffff81184ee2>] ? vfs_ioctl+0x22/0xa0
 [<ffffffff8104af98>] ? __wake_up_locked_key+0x18/0x20
 [<ffffffff811b6b93>] ? eventfd_write+0x193/0x1d0
 [<ffffffff811853aa>] ? do_vfs_ioctl+0x3aa/0x580
 [<ffffffffa017ad53>] ? kvm_on_user_return+0x73/0x80 [kvm]
 [<ffffffff81185601>] ? sys_ioctl+0x81/0xa0
 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b