Multiple invocations to NMI crash shootdown when crashing cause a list_add double add error and inability to capture a vmcore file due to a missing backport of 26044aff37a5

Issue

• Multiple invocations to NMI crash shootdown when crashing cause a list_add double add error and inability to capture a vmcore file due to a missing backport of 26044aff37a5

[926344.782067] kernel BUG at mm/slub.c:373!
[926344.784624] invalid opcode: 0000 [#1] SMP NOPTI
[926344.786859] CPU: 18 PID: 260957 Comm: mcast-receiver- Tainted: G           OE    --------- -  - 4.18.0-372.82.1.el8_6.x86_64 #1
[926344.792390] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 11/28/2023
[926344.797736] RIP: 0010:__slab_free+0x19e/0x340
[926344.799896] Code: 1f 44 00 00 eb 9c 41 f7 46 08 00 0d 21 00 0f 85 16 ff ff ff 4d 85 ed 0f 85 0d ff ff ff 80 4c 24 5b 80 45 31 ff e9 57 ff ff ff <0f> 0b 49 3b 54 24 28 75 c4 49 89 5c 24 20 49 89 4c 24 28 49 0f ba
[926344.809020] RSP: 0018:ffffa64b9fca3c60 EFLAGS: 00010246
[926344.811587] RAX: ffff954ccc717400 RBX: ffff954ccc717000 RCX: ffff954ccc717000
[926344.815053] RDX: 000000008010000f RSI: fffff5659031c400 RDI: ffff9549c0005500
[926344.818572] RBP: ffffa64b9fca3cf8 R08: 0000000000000001 R09: ffffffff9c1f75d7
[926344.822144] R10: ffff954ccc717000 R11: 0000000000000001 R12: fffff5659031c400
[926344.825659] R13: ffff954ccc717000 R14: ffff9549c0005500 R15: 0000000000000244
[926344.829100] FS:  00007f5a9cde5700(0000) GS:ffff958899c80000(0000) knlGS:0000000000000000
[926344.833268] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[926344.836037] CR2: 00007f33e92c8fd8 CR3: 0000001cea80a006 CR4: 0000000000370ee0
[926344.839420] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[926344.842909] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[926344.846376] Call Trace:
[926344.847637]  ? f_sys_recvfrom_x+0x69/0x110 [sysdigcloud_probe]
[926344.850622]  ? __skb_try_recv_from_queue+0x17a/0x1b0
[926344.853033]  ? __consume_stateless_skb+0x17/0x60
[926344.855230]  kfree+0x1d8/0x200
[926344.856720]  __consume_stateless_skb+0x17/0x60
[926344.858838]  udpv6_recvmsg+0x2b2/0x770
[926344.860641]  inet6_recvmsg+0x5b/0xd0
[926344.862433]  __sys_recvfrom+0xc3/0x180
[926344.864320]  ? syscall_trace_enter+0x1ff/0x2d0
[926344.866419]  __x64_sys_recvfrom+0x24/0x30
[926344.868322]  do_syscall_64+0x5b/0x1b0
[926344.870071]  entry_SYSCALL_64_after_hwframe+0x61/0xc6
[926344.872516] RIP: 0033:0x7f5b355a7dfa
[926344.874230] Code: 48 83 ec 18 4c 89 4c 24 08 e8 42 f9 ff ff 4c 8b 4c 24 08 4d 89 f8 45 89 f2 89 c5 4c 89 ea 4c 89 e6 89 df b8 2d 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 3a 89 ef 48 89 44 24 08 e8 72 f9 ff ff 48 8b
[926344.882919] RSP: 002b:00007f5a9cdd4590 EFLAGS: 00000293 ORIG_RAX: 000000000000002d
[926344.886424] RAX: ffffffffffffffda RBX: 000000000000002b RCX: 00007f5b355a7dfa
[926344.889733] RDX: 0000000000010000 RSI: 00007f5a28010fb0 RDI: 000000000000002b
[926344.893355] RBP: 0000000000000000 R08: 00007f5a9cdd46b0 R09: 00007f5a9cdd46a8
[926344.896693] R10: 0000000000000000 R11: 0000000000000293 R12: 00007f5a28010fb0
[926344.900554] R13: 0000000000010000 R14: 0000000000000000 R15: 00007f5a9cdd46b0
[926344.904399] Modules linked in: tcp_diag udp_diag inet_diag rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache md4 cmac nls_utf8 cifs libarc4 dns_resolver unix_diag sysdigcloud_probe(OE) binfmt_misc veth xt_addrtype nf_conntrack_netlink ipt_REJECT nf_reject_ipv4 xt_nat xt_CT xt_LOG nf_log_syslog ip6t_MASQUERADE ipt_MASQUERADE xt_mark xt_conntrack xt_comment nft_compat nft_counter nft_chain_nat nf_tables overlay geneve ip6_udp_tunnel udp_tunnel nfnetlink_cttimeout nfnetlink openvswitch nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ext4 mbcache jbd2 rpcrdma intel_rapl_msr intel_rapl_common nfit sunrpc libnvdimm kvm_intel kvm irqbypass rapl hyperv_fb hv_balloon hv_utils pcspkr joydev ip_tables xfs libcrc32c rdma_ucm ib_srpt ib_isert iscsi_target_mod target_core_mod dm_multipath ib_iser ib_umad rdma_cm ib_ipoib iw_cm ib_cm mlx5_ib ib_uverbs ib_core mlx5_core psample mlxfw pci_hyperv pci_hyperv_intf sd_mod t10_pi sg hv_storvsc serio_raw hv_netvsc scsi_transport_fc
[926344.904468]  hid_hyperv hyperv_keyboard crct10dif_pclmul crc32_pclmul crc32c_intel hv_vmbus ghash_clmulni_intel dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi ipmi_devintf scsi_transport_iscsi ipmi_msghandler fuse
[926344.963387] Red Hat flags: eBPF/event
[926344.965836] ---[ end trace 4ebf0e1f13203f47 ]---
[926344.969202] RIP: 0010:__slab_free+0x19e/0x340
[926344.972350] Code: 1f 44 00 00 eb 9c 41 f7 46 08 00 0d 21 00 0f 85 16 ff ff ff 4d 85 ed 0f 85 0d ff ff ff 80 4c 24 5b 80 45 31 ff e9 57 ff ff ff <0f> 0b 49 3b 54 24 28 75 c4 49 89 5c 24 20 49 89 4c 24 28 49 0f ba
[926344.983278] RSP: 0018:ffffa64b9fca3c60 EFLAGS: 00010246
[926344.987226] RAX: ffff954ccc717400 RBX: ffff954ccc717000 RCX: ffff954ccc717000
[926344.991923] RDX: 000000008010000f RSI: fffff5659031c400 RDI: ffff9549c0005500
[926344.996692] RBP: ffffa64b9fca3cf8 R08: 0000000000000001 R09: ffffffff9c1f75d7
[926345.001229] R10: ffff954ccc717000 R11: 0000000000000001 R12: fffff5659031c400
[926345.006283] R13: ffff954ccc717000 R14: ffff9549c0005500 R15: 0000000000000244
[926345.011150] FS:  00007f5a9cde5700(0000) GS:ffff958899c80000(0000) knlGS:0000000000000000
[926345.016181] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[926345.020068] CR2: 00007f33e92c8fd8 CR3: 0000001cea80a006 CR4: 0000000000370ee0
[926345.024669] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[926345.029246] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[926345.033824] Kernel panic - not syncing: Fatal exception
[926345.038845] Kernel Offset: 0x1aa00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[926345.072368] Rebooting in 10 seconds..
[926355.075834] list_add double add: new=ffffffff9d22eb40, prev=ffffffff9d228ee8, next=ffffffff9d22eb40.
[926355.081048] ------------[ cut here ]------------
[926355.084024] kernel BUG at lib/list_debug.c:31!
[926355.086835] invalid opcode: 0000 [#2] SMP NOPTI
[926355.089899] CPU: 18 PID: 260957 Comm: mcast-receiver- Tainted: G      D    OE    --------- -  - 4.18.0-372.82.1.el8_6.x86_64 #1
[926355.096930] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 11/28/2023
[926355.102690] RIP: 0010:__list_add_valid+0x45/0x50
[926355.106015] Code: 00 48 39 c7 74 0f 48 39 d7 74 0a b8 01 00 00 00 e9 40 6d 72 00 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 60 82 b2 9c e8 5f 9f c8 ff <0f> 0b 66 0f 1f 84 00 00 00 00 00 48 8b 07 48 8b 57 08 48 b9 00 01
[926355.116195] RSP: 0018:ffffa64b9fca3a00 EFLAGS: 00010046
[926355.119500] RAX: 0000000000000058 RBX: ffffffff9d22eb40 RCX: 0000000000000007
[926355.123552] RDX: 0000000000000000 RSI: 00000000ffffc69e RDI: ffff958899c96790
[926355.127809] RBP: 0000000000000008 R08: 0000000000000000 R09: c0000000ffffc69e
[926355.131896] R10: 000000000003f360 R11: ffffa64b9fca3820 R12: ffffffff9d228ee8
[926355.136166] R13: ffffffff9d228ee0 R14: 0000000000000046 R15: ffffffff9d22eb40
[926355.140191] FS:  00007f5a9cde5700(0000) GS:ffff958899c80000(0000) knlGS:0000000000000000
[926355.144729] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[926355.148167] CR2: 00007f33e92c8fd8 CR3: 0000001cea80a006 CR4: 0000000000370ee0
[926355.152201] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[926355.156276] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[926355.160254] Call Trace:
[926355.162182]  __register_nmi_handler+0xde/0x120
[926355.165191]  nmi_shootdown_cpus+0x3f/0xa0
[926355.167803]  native_machine_emergency_restart+0x224/0x280
[926355.171015]  panic+0x242/0x2ac
[926355.173152]  oops_end.cold.10+0x18/0x18
[926355.175670]  do_trap+0xe7/0x110
[926355.177809]  ? __slab_free+0x19e/0x340
[926355.180191]  do_invalid_op+0x36/0x40
[926355.182497]  ? __slab_free+0x19e/0x340
[926355.185281]  invalid_op+0x14/0x20
[926355.187486] RIP: 0010:__slab_free+0x19e/0x340
[926355.190114] Code: 1f 44 00 00 eb 9c 41 f7 46 08 00 0d 21 00 0f 85 16 ff ff ff 4d 85 ed 0f 85 0d ff ff ff 80 4c 24 5b 80 45 31 ff e9 57 ff ff ff <0f> 0b 49 3b 54 24 28 75 c4 49 89 5c 24 20 49 89 4c 24 28 49 0f ba
[926355.199878] RSP: 0018:ffffa64b9fca3c60 EFLAGS: 00010246
[926355.202926] RAX: ffff954ccc717400 RBX: ffff954ccc717000 RCX: ffff954ccc717000
[926355.206971] RDX: 000000008010000f RSI: fffff5659031c400 RDI: ffff9549c0005500
[926355.210842] RBP: ffffa64b9fca3cf8 R08: 0000000000000001 R09: ffffffff9c1f75d7
[926355.215030] R10: ffff954ccc717000 R11: 0000000000000001 R12: fffff5659031c400
[926355.219138] R13: ffff954ccc717000 R14: ffff9549c0005500 R15: 0000000000000244
[926355.223080]  ? __consume_stateless_skb+0x17/0x60
[926355.225918]  ? f_sys_recvfrom_x+0x69/0x110 [sysdigcloud_probe]
[926355.229213]  ? __skb_try_recv_from_queue+0x17a/0x1b0
[926355.232120]  ? __consume_stateless_skb+0x17/0x60
[926355.234933]  kfree+0x1d8/0x200
[926355.237043]  __consume_stateless_skb+0x17/0x60
[926355.239784]  udpv6_recvmsg+0x2b2/0x770
[926355.242153]  inet6_recvmsg+0x5b/0xd0
[926355.244597]  __sys_recvfrom+0xc3/0x180
[926355.247010]  ? syscall_trace_enter+0x1ff/0x2d0
[926355.249786]  __x64_sys_recvfrom+0x24/0x30
[926355.252319]  do_syscall_64+0x5b/0x1b0
[926355.254657]  entry_SYSCALL_64_after_hwframe+0x61/0xc6
[926355.257569] RIP: 0033:0x7f5b355a7dfa
[926355.259773] Code: 48 83 ec 18 4c 89 4c 24 08 e8 42 f9 ff ff 4c 8b 4c 24 08 4d 89 f8 45 89 f2 89 c5 4c 89 ea 4c 89 e6 89 df b8 2d 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 3a 89 ef 48 89 44 24 08 e8 72 f9 ff ff 48 8b
[926355.269280] RSP: 002b:00007f5a9cdd4590 EFLAGS: 00000293 ORIG_RAX: 000000000000002d
[926355.273230] RAX: ffffffffffffffda RBX: 000000000000002b RCX: 00007f5b355a7dfa
[926355.277065] RDX: 0000000000010000 RSI: 00007f5a28010fb0 RDI: 000000000000002b
[926355.280864] RBP: 0000000000000000 R08: 00007f5a9cdd46b0 R09: 00007f5a9cdd46a8
[926355.284616] R10: 0000000000000000 R11: 0000000000000293 R12: 00007f5a28010fb0
[926355.288338] R13: 0000000000010000 R14: 0000000000000000 R15: 00007f5a9cdd46b0
[926355.292109] Modules linked in: tcp_diag udp_diag inet_diag rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache md4 cmac nls_utf8 cifs libarc4 dns_resolver unix_diag sysdigcloud_probe(OE) binfmt_misc veth xt_addrtype nf_conntrack_netlink ipt_REJECT nf_reject_ipv4 xt_nat xt_CT xt_LOG nf_log_syslog ip6t_MASQUERADE ipt_MASQUERADE xt_mark xt_conntrack xt_comment nft_compat nft_counter nft_chain_nat nf_tables overlay geneve ip6_udp_tunnel udp_tunnel nfnetlink_cttimeout nfnetlink openvswitch nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ext4 mbcache jbd2 rpcrdma intel_rapl_msr intel_rapl_common nfit sunrpc libnvdimm kvm_intel kvm irqbypass rapl hyperv_fb hv_balloon hv_utils pcspkr joydev ip_tables xfs libcrc32c rdma_ucm ib_srpt ib_isert iscsi_target_mod target_core_mod dm_multipath ib_iser ib_umad rdma_cm ib_ipoib iw_cm ib_cm mlx5_ib ib_uverbs ib_core mlx5_core psample mlxfw pci_hyperv pci_hyperv_intf sd_mod t10_pi sg hv_storvsc serio_raw hv_netvsc scsi_transport_fc
[926355.292178]  hid_hyperv hyperv_keyboard crct10dif_pclmul crc32_pclmul crc32c_intel hv_vmbus ghash_clmulni_intel dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi ipmi_devintf scsi_transport_iscsi ipmi_msghandler fuse
[926355.347337] Red Hat flags: eBPF/event
[926355.349582] ---[ end trace 4ebf0e1f13203f48 ]---
[926355.352202] RIP: 0010:__slab_free+0x19e/0x340
[926355.354728] Code: 1f 44 00 00 eb 9c 41 f7 46 08 00 0d 21 00 0f 85 16 ff ff ff 4d 85 ed 0f 85 0d ff ff ff 80 4c 24 5b 80 45 31 ff e9 57 ff ff ff <0f> 0b 49 3b 54 24 28 75 c4 49 89 5c 24 20 49 89 4c 24 28 49 0f ba
[926355.364064] RSP: 0018:ffffa64b9fca3c60 EFLAGS: 00010246
[926355.367166] RAX: ffff954ccc717400 RBX: ffff954ccc717000 RCX: ffff954ccc717000
[926355.370905] RDX: 000000008010000f RSI: fffff5659031c400 RDI: ffff9549c0005500
[926355.374570] RBP: ffffa64b9fca3cf8 R08: 0000000000000001 R09: ffffffff9c1f75d7
[926355.378294] R10: ffff954ccc717000 R11: 0000000000000001 R12: fffff5659031c400
[926355.381939] R13: ffff954ccc717000 R14: ffff9549c0005500 R15: 0000000000000244
[926355.385652] FS:  00007f5a9cde5700(0000) GS:ffff958899c80000(0000) knlGS:0000000000000000
[926355.389729] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[926355.392848] CR2: 00007f33e92c8fd8 CR3: 0000001cea80a006 CR4: 0000000000370ee0
[926355.396605] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[926355.400268] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[926355.403916] Kernel panic - not syncing: Fatal exception
[926355.406951] Kernel Offset: 0x1aa00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[926355.412649] hyperv_fb: Unable to send packet via vmbus; error -11
[926355.412650] hyperv_fb: Unable to send packet via vmbus; error -11
[926355.412651] hyperv_fb: Unable to send packet via vmbus; error -11
[926355.412651] hyperv_fb: Unable to send packet via vmbus; error -11
[926355.412652] hyperv_fb: Unable to send packet via vmbus; error -11
[926355.412652] hyperv_fb: Unable to send packet via vmbus; error -11
[926355.412652] hyperv_fb: Unable to send packet via vmbus; error -11
[926355.412653] hyperv_fb: Unable to send packet via vmbus; error -11
[926355.412653] hyperv_fb: Unable to send packet via vmbus; error -11
[926355.412654] hyperv_fb: Unable to send packet via vmbus; error -11
[926355.452162] Rebooting in 10 seconds..