RHEL7: Kernel panic due to a concurrent kernel memory cache destruction race

Solution Verified - Updated -

Issue

  • Kernel panic at kmem_cache_destroy_work_func:
[39999272.660295] cni0: port 4(vetha9ac64d5) entered disabled state
[39999272.666647] device vetha9ac64d5 left promiscuous mode
[39999272.666662] cni0: port 4(vetha9ac64d5) entered disabled state
[39999272.755789] BUG: unable to handle kernel NULL pointer dereference at 00000000000000b8
[39999272.755794] IP: [<ffffffff811edbb1>] kmem_cache_destroy_work_func+0x31/0x70
[39999272.755800] PGD 0 
[39999272.755802] Oops: 0000 [#1] SMP 
[39999272.755804] Modules linked in: ScvTOS50(OE) ScvTOS50Hook(OE) xt_recent xt_physdev veth nf_tables(T) vxlan ip6_udp_tunnel udp_tunnel xt_statistic xt_nat ipt_REJECT nf_reject_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_mangle xt_comment xt_mark nfsv3 binfmt_misc fuse xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache overlay(T) vmw_vsock_vmci_transport vsock ppdev vmw_balloon intel_powerclamp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg i2c_piix4 vmw_vmci shpchp nfit libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace
[39999272.755834]  sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptsas scsi_transport_sas vmxnet3 ata_piix drm mptscsih libata e1000 mptbase i2c_core floppy fjes dm_mirror dm_region_hash dm_log dm_mod
[39999272.755850] CPU: 4 PID: 18401 Comm: kworker/4:0 Tainted: G           OE  ------------ T 3.10.0-514.el7.x86_64 #1
[39999272.755851] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[39999272.755854] Workqueue: events kmem_cache_destroy_work_func
[39999272.755855] task: ffff881f612d2f10 ti: ffff88038ca78000 task.ti: ffff88038ca78000
[39999272.755856] RIP: 0010:[<ffffffff811edbb1>]  [<ffffffff811edbb1>] kmem_cache_destroy_work_func+0x31/0x70
[39999272.755859] RSP: 0018:ffff88038ca7be10  EFLAGS: 00010286
[39999272.755859] RAX: 000000000000011b RBX: 0000000000000000 RCX: ffff881c6ec04000
[39999272.755860] RDX: ffff881fa7b66800 RSI: 3a80000000000000 RDI: ffff881f02b0c750
[39999272.755861] RBP: ffff88038ca7be18 R08: ffff881f02b0c758 R09: df837d5be570c750
[39999272.755862] R10: df837d5be570c750 R11: ffffea007ea99e80 R12: ffff88026573a200
[39999272.755862] R13: ffff88203fd16480 R14: ffff88203fd1a700 R15: 0000000000000100
[39999272.755864] FS:  0000000000000000(0000) GS:ffff88203fd00000(0000) knlGS:0000000000000000
[39999272.755865] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[39999272.755866] CR2: 00000000000000b8 CR3: 00000001276ee000 CR4: 00000000003407e0
[39999272.755890] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[39999272.755899] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[39999272.755900] Stack:
[39999272.755900]  ffff881f02b0c750 ffff88038ca7be60 ffffffff810a7f3b 000000003fd164a0
[39999272.755902]  0000000000000000 ffff88203fd164a0 ffff88026573a230 ffff881f612d2f10
[39999272.755903]  ffff88026573a200 ffff88203fd16480 ffff88038ca7bec0 ffffffff810a8d76
[39999272.755905] Call Trace:
[39999272.755909]  [<ffffffff810a7f3b>] process_one_work+0x17b/0x470
[39999272.755910]  [<ffffffff810a8d76>] worker_thread+0x126/0x410
[39999272.755912]  [<ffffffff810a8c50>] ? rescuer_thread+0x460/0x460
[39999272.755915]  [<ffffffff810b052f>] kthread+0xcf/0xe0
[39999272.755917]  [<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
[39999272.755921]  [<ffffffff81696418>] ret_from_fork+0x58/0x90
[39999272.755922]  [<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
[39999272.755923] Code: 48 89 e5 53 48 8b 57 d8 48 8b 47 f0 48 85 d2 48 8b 88 b8 00 00 00 48 c7 c0 ff ff ff ff 74 07 48 63 82 40 03 00 00 48 8b 5c c1 08 <48> 8b 83 b8 00 00 00 48 89 df 8b 40 2c 85 c0 75 0e e8 f9 7a fb 
[39999272.755939] RIP  [<ffffffff811edbb1>] kmem_cache_destroy_work_func+0x31/0x70
[39999272.755941]  RSP <ffff88038ca7be10>
[39999272.755941] CR2: 00000000000000b8

Environment

  • Red Hat Enterprise Linux 7

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content