Server crashed with Kernel panic - not syncing: Hard LOCKUP due to third party module mpt3sas.

Issue

• The kernel ring buffer was filled with the following mpt3sas and SCSI messages before the hard lockup detector triggered the panic string.

    [238049.439823] mpt3sas_cm0:    break from _scsih_search_responding_sas_devices: ioc_status(0x0022), loginfo(0x310f0400)
    [238049.439824] mpt3sas_cm0: search for end-devices: complete
    [238049.439824] mpt3sas_cm0: search for end-devices: start
    [238049.439824] mpt3sas_cm0: search for PCIe end-devices: complete
    [238049.439825] mpt3sas_cm0: search for expanders: start
    [238049.439858]         expander present: handle(0x0009), sas_addr(0x500056b30d37bdff), port:255
    [238049.439923] mpt3sas_cm0:    break from _scsih_search_responding_expanders: ioc_status(0x0022), loginfo(0x310f0400)
    [238049.439924] mpt3sas_cm0: search for expanders: complete
    [238049.439927] mpt3sas_cm0: _base_fault_reset_work: hard reset: success
    [238049.439930] mpt3sas_cm0: removing unresponding devices: start
    [238049.439930] mpt3sas_cm0: removing unresponding devices: sas end-devices
    [238049.439931] mpt3sas_cm0: removing unresponding devices: pcie end-devices
    [238049.439932] mpt3sas_cm0: removing unresponding devices: expanders
    [238049.439932] mpt3sas_cm0: removing unresponding devices: complete
    [238049.439936] mpt3sas_cm0: TEST_UNIT_READY: handle(0x000d), lun(0)
    [238050.438218] mpt3sas_cm0: TEST_UNIT_READY: handle(0x000d), lun(0)
    [238050.438358] sd 17:0:3:0: device_unblock and setting to running, handle(0x000d)
    [238050.438360] sd 17:0:3:0: device_unblocked, handle(0x000d)
    [238050.438365] mpt3sas_cm0: TEST_UNIT_READY: handle(0x0015), lun(0)
    [238051.436841] mpt3sas_cm0: TEST_UNIT_READY: handle(0x0015), lun(0)
    [238051.436985] sd 17:0:11:0: device_unblock and setting to running, handle(0x0015)
    [238051.436987] sd 17:0:11:0: device_unblocked, handle(0x0015)
    [238051.436992] mpt3sas_cm0: scan devices: start
    [238051.437284] mpt3sas_cm0:    scan devices: expanders start
    [238051.439090] mpt3sas_cm0:    break from expander scan: ioc_status(0x0022), loginfo(0x310f0400)
    [238051.439090] mpt3sas_cm0:    scan devices: expanders complete
    [238051.439091] mpt3sas_cm0:    scan devices: sas end devices start
    [238051.440034] mpt3sas_cm0:    break from sas end device scan: ioc_status(0x0022), loginfo(0x310f0400)
    [238051.440034] mpt3sas_cm0:    scan devices: sas end devices complete
    [238051.440034] mpt3sas_cm0:    scan devices: pcie end devices start
    [238051.440047] mpt3sas_cm0: log_info(0x3003011d): originator(IOP), code(0x03), sub_code(0x011d)
    [238051.440060] mpt3sas_cm0: log_info(0x3003011d): originator(IOP), code(0x03), sub_code(0x011d)
    [238051.440062] mpt3sas_cm0:    break from pcie end device scan: ioc_status(0x0021), loginfo(0x3003011d)
    [238051.440062] mpt3sas_cm0:    pcie devices: pcie end devices complete
    [238051.440063] mpt3sas_cm0: scan devices: complete
    [238074.477096] NMI watchdog: Watchdog detected hard LOCKUP on cpu 29