RHEL 8.4: LIST_POISON2 で list_del が破損したため、カーネルがクラッシュしました

Issue

• この問題は、kernel-4.18.0-240.22.1.el8 から kernel-4.18.0-305.el8 へのカーネルアップグレードで発生し始めました。
• カーネルが次のログでクラッシュしました:

[570928.662632] list_del corruption, ffff8b3c3b76b048->prev is LIST_POISON2 (dead000000000200)
[570928.662739] ------------[ cut here ]------------
[570928.662740] kernel BUG at lib/list_debug.c:50!
[570928.662773] invalid opcode: 0000 [#1] SMP PTI
[570928.662790] CPU: 2 PID: 756280 Comm: kworker/2:0 Kdump: loaded Not tainted 4.18.0-305.el8.x86_64 #1
[570928.662818] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[570928.662853] Workqueue: cgroup_destroy css_release_work_fn
[570928.662874] RIP: 0010:__list_del_entry_valid.cold.1+0x45/0x4c
[570928.662894] Code: e8 8a a5 cb ff 0f 0b 48 89 f2 48 89 fe 48 c7 c7 40 66 10 95 e8 76 a5 cb ff 0f 0b 48 89 fe 48 c7 c7 08 66 10 95 e8 65 a5 cb ff <0f> 0b 90 90 90 90 90 41 55 41 54 55 53 48 85 d2 74 5f 48 85 f6 74
[570928.662950] RSP: 0018:ffffa22203613e68 EFLAGS: 00010246
[570928.662969] RAX: 000000000000004e RBX: ffff8b3c3b76b090 RCX: 0000000000000000
[570928.662992] RDX: 0000000000000000 RSI: ffff8b3f33d167c8 RDI: ffff8b3f33d167c8
[570928.663014] RBP: ffffffff95826040 R08: 00000000000005b7 R09: 0000000000aaaaaa
[570928.663037] R10: 0000000000000000 R11: ffffa22202dff200 R12: ffff8b3c3b76b000
[570928.663059] R13: ffff8b3f2c0b0000 R14: ffff8b3dfe60d240 R15: ffff8b3c3b76b098
[570928.663082] FS:  0000000000000000(0000) GS:ffff8b3f33d00000(0000) knlGS:0000000000000000
[570928.663107] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[570928.663126] CR2: 00007f2a5bddc500 CR3: 00000001e1a10005 CR4: 00000000003706e0
[570928.663184] Call Trace:
[570928.663204]  css_release_work_fn+0x3f/0x240
[570928.663254]  process_one_work+0x1a7/0x360
[570928.663276]  worker_thread+0x30/0x390
[570928.663291]  ? create_worker+0x1a0/0x1a0
[570928.663305]  kthread+0x116/0x130
[570928.663326]  ? kthread_flush_work_fn+0x10/0x10
[570928.663344]  ret_from_fork+0x35/0x40
[570928.663361] Modules linked in: binfmt_misc nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_tables_set nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock intel_rapl_msr intel_rapl_common sb_edac crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl vmw_balloon joydev pcspkr i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic vmwgfx sd_mod t10_pi sg drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel ata_piix ahci libahci serio_raw libata vmxnet3 vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod fuse

• ログの別のパターン:

[1202369.537819] list_del corruption. next->prev should be ffff94c97f3f2098, but was ffff94c49aa94ad8
[1202369.538099] ------------[ cut here ]------------
[1202369.538221] kernel BUG at lib/list_debug.c:56!
[1202369.538418] invalid opcode: 0000 [#1] SMP PTI
[1202369.538539] CPU: 5 PID: 883812 Comm: kworker/5:1 Kdump: loaded Not tainted 4.18.0-305.3.1.el8_4.x86_64 #1
[1202369.538689] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[1202369.538873] Workqueue: 0x0 (cgroup_destroy)
[1202369.539025] RIP: 0010:__list_del_entry_valid.cold.1+0x20/0x4c
[1202369.539178] Code: 65 10 93 e8 dc a4 cb ff 0f 0b 48 89 fe 48 89 c2 48 c7 c7 18 66 10 93 e8 c8 a4 cb ff 0f 0b 48 c7 c7 c8 66 10 93 e8 ba a4 cb ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 88 66 10 93 e8 a6 a4 cb ff 0f 0b
[1202369.539488] RSP: 0018:ffffb32bd004fe60 EFLAGS: 00010046
[1202369.539622] RAX: 0000000000000054 RBX: ffff94c97f3f2098 RCX: 0000000000000000
[1202369.539769] RDX: 0000000000000000 RSI: ffff94c9a5f567c8 RDI: ffff94c9a5f567c8
[1202369.539912] RBP: ffff94c97f3f2090 R08: 00000000000006d8 R09: 0000000000aaaaaa
[1202369.540052] R10: 0000000000000000 R11: ffffb32bc4f40200 R12: ffff94c4ca311090
[1202369.540195] R13: ffff94c9a5f697e0 R14: ffffffff920fea30 R15: 0000000000000000
[1202369.540347] FS: 0000000000000000(0000) GS:ffff94c9a5f40000(0000) knlGS:0000000000000000
[1202369.540493] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1202369.540635] CR2: 00000000000000b0 CR3: 0000000408810003 CR4: 00000000001706e0
[1202369.540805] Call Trace:
[1202369.540942] move_linked_works+0x49/0xa0
[1202369.541071] ? create_worker+0x1a0/0x1a0
[1202369.541190] pwq_activate_delayed_work+0x3e/0xb0
[1202369.541327] pwq_dec_nr_in_flight+0x5d/0x90
[1202369.541459] worker_thread+0x30/0x390
[1202369.541573] ? create_worker+0x1a0/0x1a0
[1202369.541724] kthread+0x116/0x130
[1202369.541849] ? kthread_flush_work_fn+0x10/0x10
[1202369.541995] ret_from_fork+0x35/0x40