Soft lockup in multi_cpu_stop+0x7f

Issue

• Server hangs with following soft lockup error messages:

kernel: NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [migration/2:18]
kernel: Modules linked in: binfmt_misc xt_addrtype br_netfilter dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock intel_powerclamp coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr sg parport_pc
kernel: shpchp parport i2c_piix4 vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crct10dif_common ttm crc32c_intel serio_raw drm ata_piix vmxnet3 libata vmw_pvscsi i2c_core floppy fjes dm_mirror dm_region_hash dm_log dm_mod
kernel: CPU: 2 PID: 18 Comm: migration/2 Not tainted 3.10.0-514.el7.x86_64 #1
kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012
kernel: task: ffff880184d43ec0 ti: ffff880184d70000 task.ti: ffff880184d70000
kernel: RIP: 0010:[<ffffffff8111613f>]  [<ffffffff8111613f>] multi_cpu_stop+0x7f/0xf0
kernel: RSP: 0000:ffff880184d73da0  EFLAGS: 00000293
kernel: RAX: ffffffff816b19a0 RBX: ffff88023529ce70 RCX: dead000000000200
kernel: RDX: 0000000000000001 RSI: 0000000000000286 RDI: ffff880343203b70
kernel: RBP: ffff880184d73dc0 R08: 0000000000000000 R09: 0000000000000000
kernel: R10: 000000000000bb34 R11: 0000000000000001 R12: 0000000084d73d38
kernel: R13: ffffffff810bcf71 R14: ffff880184d73d28 R15: ffffffff811160c0
kernel: FS:  0000000000000000(0000) GS:ffff880237d00000(0000) knlGS:0000000000000000
kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 00007fe0acc57960 CR3: 00000000019ba000 CR4: 00000000000407e0
kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
kernel: Stack:
kernel: ffff880237d0ff08 ffff880237d0ff00 ffff880343203b70 ffff880343203b98
kernel: ffff880184d73e90 ffffffff811163c6 ffff880184d73fd8 ffff880237d0ff10
kernel: 0000000000000000 0000000000000000 ffff880237d16c40 ffff88023696b200
kernel: Call Trace:
kernel: [<ffffffff811163c6>] cpu_stopper_thread+0x96/0x170
kernel: [<ffffffff8168af70>] ? __schedule+0x3b0/0x990
kernel: [<ffffffff810b90af>] smpboot_thread_fn+0x12f/0x180
kernel: [<ffffffff810b8f80>] ? lg_double_unlock+0x90/0x90
kernel: [<ffffffff810b052f>] kthread+0xcf/0xe0
kernel: [<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
kernel: [<ffffffff81696418>] ret_from_fork+0x58/0x90
kernel: [<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
kernel: Code: 75 05 45 84 f6 75 65 f0 ff 4b 24 0f 94 c1 84 c9 44 89 e2 74 0f 8b 43 20 8b 73 10 8d 48 01 89 73 24 89 4b 20 83 fa 04 74 23 f3 90 <44> 8b 63 20 41 39 d4 74 f0 41 83 fc 02 75 c2 fa 66 66 90 66 66