VMware virtual machine guest soft lockups in mpt_put_msg_frame

Issue

• VMware guest detects soft lockups
• There are hung tasks reported in the messages

grep -e "soft lockup" -e RIP messages
<time> <hostname> kernel: NMI watchdog: BUG: soft lockup - CPU#6 stuck for 25s! [ksoftirqd/6:38]
<time> <hostname> kernel: RIP: 0010:[<ffffffffa007f3be>]  [<ffffffffa007f3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
<time> <hostname> kernel: NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [ksoftirqd/7:43]
<time> <hostname> kernel: RIP: 0010:[<ffffffffa007f3be>]  [<ffffffffa007f3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
<time> <hostname> kernel: NMI watchdog: BUG: soft lockup - CPU#7 stuck for 24s! [NetworkManager:945]
<time> <hostname> kernel: RIP: 0010:[<ffffffffa007f3be>]  [<ffffffffa007f3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
<time> <hostname> kernel: NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [watchdog/7:41]
...

• If softlockup_panic kernel tunable is enabled, the VMware guest crashes with following kernel panic stack trace

crash> softlockup_panic
softlockup_panic = $2 = 0x1    <--- softlockup_panic is enabled, hence the explicit panic call

crash> bt
PID: 43     TASK: ffff880153468fb0  CPU: 7   COMMAND: "ksoftirqd/7"
 #0 [ffff88203fdc3cf0] machine_kexec at ffffffff81059beb
 #1 [ffff88203fdc3d50] __crash_kexec at ffffffff81105822
 #2 [ffff88203fdc3e20] panic at ffffffff81680441
 #3 [ffff88203fdc3ea0] watchdog_timer_fn at ffffffff8112f454
 #4 [ffff88203fdc3ed8] __hrtimer_run_queues at ffffffff810b4d72
 #5 [ffff88203fdc3f30] hrtimer_interrupt at ffffffff810b5310
 #6 [ffff88203fdc3f80] local_apic_timer_interrupt at ffffffff81051037
 #7 [ffff88203fdc3f98] smp_apic_timer_interrupt at ffffffff81699e0f
 #8 [ffff88203fdc3fb0] apic_timer_interrupt at ffffffff8169835d
--- <IRQ stack> ---
 #9 [ffff8801534739f8] apic_timer_interrupt at ffffffff8169835d
    [exception RIP: mpt_put_msg_frame+0x5e]
    RIP: ffffffffa007b3be  RSP: ffff880153473aa0  RFLAGS: 00000246
    RAX: ffffc9000c140000  RBX: ffff880153473a70  RCX: 000000000000001b
    RDX: ffff881fbae9fa00  RSI: ffff88203fda8000  RDI: 000000000000000e
    RBP: ffff880153473ab0   R8: 0000000000000003   R9: ffff88203fdaf0d8
    R10: ffff881b26273d40  R11: ffff881b26273d40  R12: 0000000100000002
    R13: ffff881a99e72480  R14: ffff880153473a14  R15: ffffffff812f53eb
    ORIG_RAX: ffffffffffffff10  CS: 0010  SS: 0018
#10 [ffff880153473ab8] mptscsih_qcmd at ffffffffa0142729 [mptscsih]
#11 [ffff880153473b60] mptspi_qcmd at ffffffffa00d42b0 [mptspi]
#12 [ffff880153473b88] scsi_dispatch_cmd at ffffffff8145259a
#13 [ffff880153473bb0] scsi_request_fn at ffffffff8145b6a1
#14 [ffff880153473c18] __blk_run_queue at ffffffff812eb9c3
#15 [ffff880153473c30] blk_run_queue at ffffffff812eba36
#16 [ffff880153473c50] scsi_run_queue at ffffffff81459a38
#17 [ffff880153473cb0] scsi_next_command at ffffffff8145b950
#18 [ffff880153473cd0] scsi_end_request at ffffffff8145baa5
#19 [ffff880153473d10] scsi_io_completion at ffffffff8145bd18
#20 [ffff880153473d70] scsi_finish_command at ffffffff81450e75
#21 [ffff880153473da0] scsi_softirq_done at ffffffff8145b172
#22 [ffff880153473dd0] blk_done_softirq at ffffffff812f6390
#23 [ffff880153473e10] __do_softirq at ffffffff8108f63f
#24 [ffff880153473e80] run_ksoftirqd at ffffffff8108f808
#25 [ffff880153473e98] smpboot_thread_fn at ffffffff810b95bf
#26 [ffff880153473ec8] kthread at ffffffff810b0a4f
#27 [ffff880153473f50] ret_from_fork at ffffffff81697658