RHEL7: Kernel panic due to a concurrent kernel memory cache destruction race
Issue
- Kernel panic at
kmem_cache_destroy_work_func
:
[39999272.660295] cni0: port 4(vetha9ac64d5) entered disabled state
[39999272.666647] device vetha9ac64d5 left promiscuous mode
[39999272.666662] cni0: port 4(vetha9ac64d5) entered disabled state
[39999272.755789] BUG: unable to handle kernel NULL pointer dereference at 00000000000000b8
[39999272.755794] IP: [<ffffffff811edbb1>] kmem_cache_destroy_work_func+0x31/0x70
[39999272.755800] PGD 0
[39999272.755802] Oops: 0000 [#1] SMP
[39999272.755804] Modules linked in: ScvTOS50(OE) ScvTOS50Hook(OE) xt_recent xt_physdev veth nf_tables(T) vxlan ip6_udp_tunnel udp_tunnel xt_statistic xt_nat ipt_REJECT nf_reject_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_mangle xt_comment xt_mark nfsv3 binfmt_misc fuse xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache overlay(T) vmw_vsock_vmci_transport vsock ppdev vmw_balloon intel_powerclamp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg i2c_piix4 vmw_vmci shpchp nfit libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace
[39999272.755834] sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptsas scsi_transport_sas vmxnet3 ata_piix drm mptscsih libata e1000 mptbase i2c_core floppy fjes dm_mirror dm_region_hash dm_log dm_mod
[39999272.755850] CPU: 4 PID: 18401 Comm: kworker/4:0 Tainted: G OE ------------ T 3.10.0-514.el7.x86_64 #1
[39999272.755851] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[39999272.755854] Workqueue: events kmem_cache_destroy_work_func
[39999272.755855] task: ffff881f612d2f10 ti: ffff88038ca78000 task.ti: ffff88038ca78000
[39999272.755856] RIP: 0010:[<ffffffff811edbb1>] [<ffffffff811edbb1>] kmem_cache_destroy_work_func+0x31/0x70
[39999272.755859] RSP: 0018:ffff88038ca7be10 EFLAGS: 00010286
[39999272.755859] RAX: 000000000000011b RBX: 0000000000000000 RCX: ffff881c6ec04000
[39999272.755860] RDX: ffff881fa7b66800 RSI: 3a80000000000000 RDI: ffff881f02b0c750
[39999272.755861] RBP: ffff88038ca7be18 R08: ffff881f02b0c758 R09: df837d5be570c750
[39999272.755862] R10: df837d5be570c750 R11: ffffea007ea99e80 R12: ffff88026573a200
[39999272.755862] R13: ffff88203fd16480 R14: ffff88203fd1a700 R15: 0000000000000100
[39999272.755864] FS: 0000000000000000(0000) GS:ffff88203fd00000(0000) knlGS:0000000000000000
[39999272.755865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[39999272.755866] CR2: 00000000000000b8 CR3: 00000001276ee000 CR4: 00000000003407e0
[39999272.755890] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[39999272.755899] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[39999272.755900] Stack:
[39999272.755900] ffff881f02b0c750 ffff88038ca7be60 ffffffff810a7f3b 000000003fd164a0
[39999272.755902] 0000000000000000 ffff88203fd164a0 ffff88026573a230 ffff881f612d2f10
[39999272.755903] ffff88026573a200 ffff88203fd16480 ffff88038ca7bec0 ffffffff810a8d76
[39999272.755905] Call Trace:
[39999272.755909] [<ffffffff810a7f3b>] process_one_work+0x17b/0x470
[39999272.755910] [<ffffffff810a8d76>] worker_thread+0x126/0x410
[39999272.755912] [<ffffffff810a8c50>] ? rescuer_thread+0x460/0x460
[39999272.755915] [<ffffffff810b052f>] kthread+0xcf/0xe0
[39999272.755917] [<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
[39999272.755921] [<ffffffff81696418>] ret_from_fork+0x58/0x90
[39999272.755922] [<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
[39999272.755923] Code: 48 89 e5 53 48 8b 57 d8 48 8b 47 f0 48 85 d2 48 8b 88 b8 00 00 00 48 c7 c0 ff ff ff ff 74 07 48 63 82 40 03 00 00 48 8b 5c c1 08 <48> 8b 83 b8 00 00 00 48 89 df 8b 40 2c 85 c0 75 0e e8 f9 7a fb
[39999272.755939] RIP [<ffffffff811edbb1>] kmem_cache_destroy_work_func+0x31/0x70
[39999272.755941] RSP <ffff88038ca7be10>
[39999272.755941] CR2: 00000000000000b8
Environment
- Red Hat Enterprise Linux 7
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.