RHEL 8.4: Kernel BUG at arch/x86/kernel/nmi.c:507!

Issue

• Kernel panic with below logs:

[50171.249303] ontuned[202757]: segfault at 7ff2366fb040 ip 00007ff246ac3184 sp 00007ff244a19718 error 6 in libc-2.28.so[7ff246a1f000+1bc000]
...
[92122.840600] NOHZ: local_softirq_pending 280
[92237.798990] ------------[ cut here ]------------
[92237.798991] kernel BUG at arch/x86/kernel/nmi.c:507!
[92237.798991] invalid opcode: 0000 [#1] SMP NOPTI
[92237.798991] CPU: 36 PID: 1005244 Comm: PIPMDMain Kdump: loaded Not tainted 4.18.0-305.12.1.el8_4.x86_64 #1
[92237.798992] Hardware name: HPE ProLiant DL380 Gen10/ProLiant DL380 Gen10, BIOS U30 01/23/2021
[92237.798992] RIP: 0010:do_nmi+0x1bc/0x1e0
[92237.798993] Code: 00 00 e8 f7 4c 04 00 66 90 e9 56 ff ff ff e8 6b 82 91 00 e9 64 fe ff ff 65 c7 05 7b fd 5e 5e 02 00 00 00 5b 5d 41 5c 41 5d c3 <0f> 0b 0f 0b 65 48 8b 3d 60 fd 5e 5e e8 93 46 04 00 66 90 e9 3f ff
[92237.798993] RSP: 0000:fffffe0000735ed0 EFLAGS: 00010006
[92237.798994] RAX: 0000000080110000 RBX: 0000000000000007 RCX: 0000000000000000
[92237.798994] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000007
[92237.798994] RBP: fffffe0000735ef8 R08: 0000000000000000 R09: 0000000000000000
[92237.798994] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000ffffffff
[92237.798994] R13: 0000000000000200 R14: 0000000000000000 R15: 0000000000000000
[92237.798995] FS:  00007ffff0d88700(0000) GS:ffff9c77bf900000(0000) knlGS:0000000000000000
[92237.798995] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[92237.798995] CR2: 0000560396f1af44 CR3: 0000002bc1bd8003 CR4: 00000000007706e0
[92237.798995] DR0: 00007ffff3481b90 DR1: 0000000000000000 DR2: 0000000000000000
[92237.798996] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[92237.798996] PKRU: 55555554
[92237.798996] Call Trace:
[92237.798996]  <NMI>
[92237.798996]  end_repeat_nmi+0x16/0x6f
[92237.798996] RIP: 0010:__sigqueue_alloc+0x10/0xa0
[92237.798997] Code: 89 c0 e9 e3 08 11 00 0f 1f 00 89 f6 e9 99 07 11 00 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 89 c8 53 48 8b 8e d0 0a 00 00 <48> 8b 99 80 00 00 00 f0 ff 03 0f 88 ca ee 84 00 f0 ff 43 08 85 c0
[92237.798997] RSP: 0000:ffffc0d8fbf73e40 EFLAGS: 00000803
[92237.798998] RAX: 0000000000000001 RBX: 0000000000000005 RCX: ffff9c63510012c0
[92237.798998] RDX: 0000000000480020 RSI: ffff9c6637888000 RDI: 0000000000000005
[92237.798998] RBP: ffff9c6637888000 R08: ffff9c6637888b58 R09: ffff9c6637888000
[92237.798998] R10: 0000000000000000 R11: 0000000000000000 R12: ffffc0d8fbf73ed0
[92237.798998] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000004
[92237.798999]  ? __sigqueue_alloc+0x10/0xa0
[92237.798999]  ? __sigqueue_alloc+0x10/0xa0
[92237.798999]  </NMI>
[92237.798999]  __send_signal+0x317/0x4a0
[92237.798999]  ? do_signal+0x36/0x660
[92237.798999]  force_sig_info+0xaf/0xe0
[92237.799000]  force_sig_fault+0x59/0x80
[92237.799000]  do_debug+0x22b/0x2f0
[92237.799000]  ? debug+0x25/0x40
[92237.799000]  debug+0x2f/0x40
[92237.799000] RIP: 0033:0x7ffff303d998
[92237.799001] Code: c1 e2 06 4c 8d 2c 2a bd 11 27 00 00 4c 8b 35 5f f6 24 00 49 8b 1e 42 83 bc 2b f4 a5 00 00 00 0f 85 59 01 00 00 bf e8 03 00 00 <e8> a3 0a ff ff 83 ed 01 75 df 66 48 8d 3d 3e f6 24 00 66 66 48 e8
[92237.799001] RSP: 002b:00007ffff0652220 EFLAGS: 00000346
[92237.799001] RAX: 0000000000000000 RBX: 00007ffff7e9100c RCX: 0000000000000000
[92237.799001] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000000003e8
[92237.799002] RBP: 00000000000006bb R08: 0000000000000000 R09: 00007ff3b3fff700
[92237.799002] R10: 00007ff3b3fff9d0 R11: 0000000000000000 R12: 0000000000000000
[92237.799002] R13: 0000000000002800 R14: 00007ffff3481b90 R15: 0000000000000001
[92237.799002] Modules linked in: xt_CHECKSUM ipt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 nft_compat nft_counter nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink tun bridge stp llc rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache bonding sunrpc dm_service_time dm_multipath intel_rapl_msr intel_rapl_common isst_if_common nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl intel_cstate intel_uncore pcspkr ipmi_ssif ses enclosure mei_me hpwdt hpilo mei lpc_ich ioatdma wmi acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter binfmt_misc ip_tables xfs libcrc32c sd_mod sr_mod cdrom sg qla2xxx mgag200 drm_kms_helper crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops serio_raw nvme_fc ixgbe drm ahci libahci nvme_fabrics smartpqi tg3 nvme_core libata scsi_transport_sas t10_pi scsi_transport_fc mdio dca i2c_algo_bit dm_mirror
[92237.799013]  dm_region_hash dm_log dm_mod fuse