System was panicked due to hard lockup post hardware errors.

Issue

• Due to faulty hardware, CPUs loop in kernel mode for longer time resulting hard lockup and causing system reboot.
• System was panicked with calltrace like below:

[Sat Jun  8 20:13:45 EDT 2024] mce: [Hardware Error]: Machine check events logged
[Sat Jun  8 22:59:46 EDT 2024] sched: RT throttling activated
[Sat Jun  8 23:00:09 EDT 2024] NMI watchdog: Watchdog detected hard LOCKUP on cpu 56
[Sat Jun  8 23:00:09 EDT 2024] Modules linked in:
[Sat Jun  8 23:00:09 EDT 2024]  falcon_lsm_serviceable(PE) falcon_nf_netcontain(E) falcon_kal(E) falcon_lsm_pinned_16703(E) unix_diag af_packet_diag netlink_diag ip6table_filter ip6_tables iptable_filter tcp_diag udp_diag inet_diag mptctl mptbase nfnetlink_queue nfnetlink_log falcon_lsm_pinned_15508(E) bonding iTCO_wdt gpio_ich iTCO_vendor_support intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel dm_service_time aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_multipath osst lpc_ich hpilo hpwdt i7core_edac ipmi_si st ipmi_devintf wmi ipmi_msghandler acpi_power_meter pcc_cpufreq auth_rpcgss sg binfmt_misc sunrpc ip_tables ext4 mbcache jbd2 lpfc sd_mod i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nvmet_fc nvmet crc_t10dif drm crct10dif_generic
[Sat Jun  8 23:00:09 EDT 2024]  nvme_fc nvme_fabrics tg3 crct10dif_pclmul nvme_core crc32c_intel hpsa serio_raw scsi_transport_fc ptp be2net scsi_transport_sas scsi_tgt drm_panel_orientation_quirks pps_core crct10dif_common dm_mirror dm_region_hash dm_log dm_mod br_netfilter bridge stp llc [last unloaded: falcon_kal]
[Sat Jun  8 23:00:09 EDT 2024] CPU: 56 PID: 63564 Comm: kworker/56:0 Kdump: loaded Tainted: P            E  ------------ T 3.10.0-1160.102.1.el7.x86_64 #1
[Sat Jun  8 23:00:09 EDT 2024] Hardware name: HP ProLiant DL580 G7, BIOS P65 10/01/2013
[Sat Jun  8 23:00:09 EDT 2024] Workqueue: events vmstat_shepherd
[Sat Jun  8 23:00:09 EDT 2024] task: ffff9e0a94a59080 ti: ffff9e06354e8000 task.ti: ffff9e06354e8000
[Sat Jun  8 23:00:09 EDT 2024] RIP: 0010:[<ffffffff9cee0d72>]  [<ffffffff9cee0d72>] try_to_wake_up+0x72/0x3a0
[Sat Jun  8 23:00:09 EDT 2024] RSP: 0000:ffff9e06354ebcb8  EFLAGS: 00000002
[Sat Jun  8 23:00:09 EDT 2024] RAX: 0000000000000001 RBX: ffff9dfc9a1c9884 RCX: 0000000000000000
[Sat Jun  8 23:00:09 EDT 2024] RDX: 0000000000000001 RSI: 0000000000000003 RDI: ffff9dfc9a1c9884
[Sat Jun  8 23:00:09 EDT 2024] RBP: ffff9e06354ebcf8 R08: ffff9dfc9f99a560 R09: ffff9dfc9f99fe88
[Sat Jun  8 23:00:09 EDT 2024] R10: 0000000000000006 R11: 0000000000000005 R12: 0000000000000000
[Sat Jun  8 23:00:09 EDT 2024] R13: ffff9dfc9a1c9080 R14: 0000000000000000 R15: 0000000000000003
[Sat Jun  8 23:00:09 EDT 2024] FS:  0000000000000000(0000) GS:ffff9e0c9f200000(0000) knlGS:0000000000000000
[Sat Jun  8 23:00:09 EDT 2024] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Sat Jun  8 23:00:09 EDT 2024] CR2: 00002b491407a350 CR3: 000000181b1ac000 CR4: 00000000000207e0
[Sat Jun  8 23:00:09 EDT 2024] Call Trace:
[Sat Jun  8 23:00:09 EDT 2024]  [<ffffffff9cee10b5>] wake_up_process+0x15/0x20
[Sat Jun  8 23:00:09 EDT 2024]  [<ffffffff9cec05b4>] wake_up_worker+0x24/0x30
[Sat Jun  8 23:00:09 EDT 2024]  [<ffffffff9cec1025>] insert_work+0x65/0xb0
[Sat Jun  8 23:00:09 EDT 2024]  [<ffffffff9cec11ac>] __queue_work+0x13c/0x400
[Sat Jun  8 23:00:09 EDT 2024]  [<ffffffff9cec156a>] __queue_delayed_work+0xaa/0x1a0
[Sat Jun  8 23:00:09 EDT 2024]  [<ffffffff9cfea6d6>] ? next_online_pgdat+0x26/0x60
[Sat Jun  8 23:00:09 EDT 2024]  [<ffffffff9cec1825>] queue_delayed_work_on+0x45/0x50
[Sat Jun  8 23:00:09 EDT 2024]  [<ffffffff9cfebabd>] vmstat_shepherd+0x8d/0xe0
[Sat Jun  8 23:00:09 EDT 2024]  [<ffffffff9cec32ef>] process_one_work+0x17f/0x440
[Sat Jun  8 23:00:09 EDT 2024]  [<ffffffff9cec4436>] worker_thread+0x126/0x3c0
[Sat Jun  8 23:00:09 EDT 2024]  [<ffffffff9cec4310>] ? manage_workers.isra.26+0x2b0/0x2b0
[Sat Jun  8 23:00:09 EDT 2024]  [<ffffffff9cecb621>] kthread+0xd1/0xe0
[Sat Jun  8 23:00:09 EDT 2024]  [<ffffffff9cecb550>] ? insert_kthread_work+0x40/0x40
[Sat Jun  8 23:00:09 EDT 2024]  [<ffffffff9d5c51f7>] ret_from_fork_nospec_begin+0x21/0x21
[Sat Jun  8 23:00:09 EDT 2024]  [<ffffffff9cecb550>] ? insert_kthread_work+0x40/0x40
[Sat Jun  8 23:00:09 EDT 2024] Code: 40 18 89 45 c0 41 8b 4d 4c 85 c9 0f 85 78 01 00 00 48 c7 45 c8 c0 ac 01 00 41 8b 55 28 85 d2 74 12 0f 1f 84 00 00 00 00 00 f3 90 <41> 8b 45 28 85 c0 75 f6 49 8b 45 00 31 d2 a8 02 74 0b 41 0f b7
[Sat Jun  8 23:00:09 EDT 2024] Kernel panic - not syncing: Hard LOCKUP