[RHEL-7.2] BUG: soft lockup - CPU#11 stuck for 23s! [migration/xx:yyy]

Issue

Migration thread soft lockup messages present on server and CPU hang

• x86_64

[  392.666057] BUG: soft lockup - CPU#31 stuck for 22s! [migration/31:420]\ <<<<<<<<<<<<<<<<<<
[  392.666057] Modules linked in: isofs adm1021 lm90 nouveau mxm_wmi wmi video ppdev           i2c_algo_bit ttm intel_rapl parport_pc parport drm_kms_helper drm i2c_piix4 crct10dif_pclmul   crct10dif_common crc32_pclmul ghash_clmulni_intel i2c_core a
esni_intel lrw gf128mul pcspkr glue_helper ablk_helper serio_raw cryptd xfs libcrc32c          ata_generic pata_acpi xen_netfront xen_blkfront ata_piix libata crc32c_intel floppy
[  392.666057] CPU: 31 PID: 420 Comm: migration/31 Not tainted 3.10.0-229.el7.x86_64 #1
[  392.666057] Hardware name: Xen HVM domU, BIOS 4.2.amazon 10/16/2015
[  392.666057] task: ffff880f158c6660 ti: ffff880f15928000 task.ti: ffff880f15928000
[  392.666057] RIP: 0010:[<ffffffff8107fd74>]  [<ffffffff8107fd74>] run_timer_softirq+0x1c4/   0x320
[  392.666057] RSP: 0018:ffff880f20fe3eb8  EFLAGS: 00000282
[  392.666057] RAX: 0000000000000000 RBX: ffffffffffffff0c RCX: 000000000000001f
[  392.666057] RDX: 00000000fffd5f69 RSI: 00000000a06aa068 RDI: ffffffff81903088
[  392.666057] RBP: ffff880f20fe3ed0 R08: ffff880f20fe3e38 R09: 00000000000002e0
[  392.666057] R10: ffff880f20fefe4c R11: 0000000000000003 R12: ffff880f20fe3e28
[  392.666057] R13: ffffffff8161586d R14: ffff880f20fe3ed0 R15: 0000000000000001
[  392.666057] FS:  0000000000000000(0000) GS:ffff880f20fe0000(0000) knlGS:0000000000000000
[  392.666057] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  392.666057] CR2: 00007fa63d0a87b0 CR3: 000000000190a000 CR4: 00000000000406e0
[  392.666057] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  392.666057] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  392.666057] Stack:
[  392.666057]  ffff8807913e7c01 ffffffff81903088 0000000000000001 ffff880f20fe3f40
[  392.666057]  ffffffff81077b2f ffff880f1592bfd8 0000000a0420a040 00000000fffd5f6b
[  392.666057]  0000000000000001 ffff880f1592bfd8 ffff880f1592bfd8 000001000000001f
[  392.666057] Call Trace:
[  392.666057]  <IRQ>
[  392.666057]  [<ffffffff81077b2f>] __do_softirq+0xef/0x280
[  392.666057]  [<ffffffff816156dc>] call_softirq+0x1c/0x30
[  392.666057]  [<ffffffff81015d95>] do_softirq+0x65/0xa0
[  392.666057]  [<ffffffff81077ec5>] irq_exit+0x115/0x120
[  392.666057]  [<ffffffff813815a5>] xen_evtchn_do_upcall+0x35/0x50
[  392.666057]  [<ffffffff8161586d>] xen_hvm_callback_vector+0x6d/0x80
[  392.666057]  <EOI>
[  392.666057]  [<ffffffff810f26dd>] ? multi_cpu_stop+0x7d/0xf0
[  392.666057]  [<ffffffff810f2660>] ? cpu_stop_should_run+0x50/0x50
[  392.666057]  [<ffffffff810f28e8>] cpu_stopper_thread+0x88/0x160
[  392.666057]  [<ffffffff81608d48>] ? __schedule+0x2d8/0x7c0
[  392.666057]  [<ffffffff8109fc7f>] smpboot_thread_fn+0xff/0x1a0
[  392.666057]  [<ffffffff81609259>] ? schedule+0x29/0x70
[  392.666057]  [<ffffffff8109fb80>] ? lg_global_unlock+0xc0/0xc0
[  392.666057]  [<ffffffff8109726f>] kthread+0xcf/0xe0
[  392.666057]  [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
[  392.666057]  [<ffffffff81613cfc>] ret_from_fork+0x7c/0xb0
[  392.666057]  [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
[  392.666057] Code: 00 e9 2e 01 00 00 66 83 03 02 fb 66 66 90 66 66 90 48 8b 45 d0 65 48 33   04 25 28 00 00 00 0f 85 4f 01 00 00 48 83 c4 40 5b 41 5c <41> 5d 41 5e 41 5f 5d c3 0f 1f 40 00 4c 8b 25 09 8c 97 00 4d 85

• ppc64

[103532.492560] BUG: soft lockup - CPU#11 stuck for 23s! [migration/11:322]
[ 5078.174544] Non critical power or cooling issue cleared
[103532.492578] Modules linked in:
[103532.492581]  pseries_energy fuse btrfs raid6_pq xor vfat msdos fat xfs libcrc32c bridge    stp llc bonding uinput pseries_rng nx_crypto xprtrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt       ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ses enclosure      binfmt_misc ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_common iw_cxgb3 ib_core ib_addr ipr  cxgb3 libata mdio dm_mirror dm_region_hash dm_log dm_mod
[103532.492640] CPU: 11 PID: 322 Comm: migration/11 Not tainted 3.10.0-229.el7.ppc64 #1
[103532.492643] task: c000000f6717df50 ti: c000000f674f4000 task.ti: c000000f674f4000
[103532.492646] NIP: c000000000189118 LR: c0000000001894f8 CTR: c0000000001890a0
[103532.492648] REGS: c000000f674f7820 TRAP: 0901   Not tainted  (3.10.0-229.el7.ppc64)
[103532.492650] MSR: 8000000100009032 <SF,EE,ME,IR,DR,RI>  CR: 24000088  XER: 20000000
[103532.492656] CFAR: c000000000189190 SOFTE: 1
                GPR00: c0000000001894f8 c000000f674f7aa0 c00000000130ae00 c000000d02253640
                GPR04: c000000d02253668 c000000d02253668 0000000000000000 0000000000000000
                GPR08: c000000d02253664 0000000000000001 0000000000000001 0000000000000003
                GPR12: 0000000024000028 c000000007b26300
[103532.492675] NIP [c000000000189118] .multi_cpu_stop+0x78/0x290
[103532.492678] LR [c0000000001894f8] .cpu_stopper_thread+0xd8/0x1f0
[103532.492680] Call Trace:
[103532.492684] [c000000f674f7aa0] [c000000000113080] .complete+0xb0/0x130 (unreliable)
[103532.492687] [c000000f674f7b40] [c0000000001894f8] .cpu_stopper_thread+0xd8/0x1f0
[103532.492690] [c000000f674f7c80] [c00000000010c748] .smpboot_thread_fn+0x228/0x280
[103532.492693] [c000000f674f7d30] [c0000000000fe528] .kthread+0xe8/0xf0
[103532.492697] [c000000f674f7e30] [c00000000000a464] .ret_from_kernel_thread+0x58/0x74
[103532.492699] Instruction dump:
[103532.492701] 7d29502a 7d3ef436 7bde07e0 2fbe0000 409e0128 39400000 38c00000 391f0024
[103532.492705] 60000000 60420000 7c210b78 7c421378 <813f0020> 7f895040 2b090002 419e0068

PID: 322    TASK: c000000f6717df50  CPU: 11  COMMAND: "migration/11"
 #0 [c000000f674f75f0] .crash_ipi_callback+0x104 at c00000000004fd64
 #1 [c000000f674f7680] .die+0x354 at c000000000020a54
 #2 [c000000f674f7730] .system_reset_exception+0x5c at c000000000020dec
 #3 [c000000f674f77b0] system_reset_common+0x108 at c000000000002488
 System Reset [100] exception frame:
 R0:  c0000000001894f8    R1:  c000000f674f7aa0    R2:  c00000000130ae00
 R3:  c000000d02253640    R4:  c000000d02253668    R5:  c000000d02253668
 R6:  0000000000000000    R7:  0000000000000000    R8:  c000000d02253664
 R9:  0000000000000001    R10: 0000000000000001    R11: 0000000000000003
 R12: 0000000024000028    R13: c000000007b26300    R14: c0000000000fe440
 R15: c000000f6a907880    R16: 0000000000000000    R17: 0000000000000000
 R18: 0000000000000000    R19: 0000000000000000    R20: 0000000000000000
 R21: 0000000000000000    R22: 0000000000000000    R23: 0000000000000000
 R24: 0000000000000001    R25: c000000f674f4000    R26: c000000001ac8820
 R27: 0000000000000000    R28: 0000000000000001    R29: c0000000012ba5d8
 R30: 0000000000000000    R31: c000000d02253640
 NIP: c000000000189118    MSR: 8000000100089032    OR3: 000000000000011c
 CTR: c0000000001890a0    LR:  c0000000001894f8    XER: 0000000020000000
 CCR: 0000000024000088    MQ:  0000000000000001    DAR: 0000000000000000
 DSISR: c000000f674f7a00     Syscall Result: 0000000000000000
 #4 [c000000f674f7aa0] .multi_cpu_stop+0x78 at c000000000189118
 [Link Register] [c000000f674f7aa0] .cpu_stopper_thread at c0000000001894f8
 #5 [c000000f674f7b40] .cpu_stopper_thread+0xd8 at c0000000001894f8  (unreliable)
 #6 [c000000f674f7c80] .smpboot_thread_fn+0x228 at c00000000010c748
 #7 [c000000f674f7d30] .kthread+0xe8 at c0000000000fe528
 #8 [c000000f674f7e30] .ret_from_kernel_thread+0x58 at c00000000000a464