RHEL 8.4: LIST_POISON2 による list_del 破壊が原因でカーネルクラッシュが発生する

Issue

• この問題は、kernel-4.18.0-240.22.1.el8 から kernel-4.18.0-305.el8 へのカーネルアップグレードで発生し始めました。

• カーネルが以下のログでクラッシュしました。

  [570928.662632] list_del corruption, ffff8b3c3b76b048->prev is LIST_POISON2 (dead000000000200)
  [570928.662739] ------------[ cut here ]------------
  [570928.662740] kernel BUG at lib/list_debug.c:50!
  [570928.662773] invalid opcode: 0000 [#1] SMP PTI
  [570928.662790] CPU: 2 PID: 756280 Comm: kworker/2:0 Kdump: loaded Not tainted 4.18.0-305.el8.x86_64 #1
  [570928.662818] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
  [570928.662853] Workqueue: cgroup_destroy css_release_work_fn
  [570928.662874] RIP: 0010:__list_del_entry_valid.cold.1+0x45/0x4c
  [570928.662894] Code: e8 8a a5 cb ff 0f 0b 48 89 f2 48 89 fe 48 c7 c7 40 66 10 95 e8 76 a5 cb ff 0f 0b 48 89 fe 48 c7 c7 08 66 10 95 e8 65 a5 cb ff <0f> 0b 90 90 90 90 90 41 55 41 54 55 53 48 85 d2 74 5f 48 85 f6 74
  [570928.662950] RSP: 0018:ffffa22203613e68 EFLAGS: 00010246
  [570928.662969] RAX: 000000000000004e RBX: ffff8b3c3b76b090 RCX: 0000000000000000
  [570928.662992] RDX: 0000000000000000 RSI: ffff8b3f33d167c8 RDI: ffff8b3f33d167c8
  [570928.663014] RBP: ffffffff95826040 R08: 00000000000005b7 R09: 0000000000aaaaaa
  [570928.663037] R10: 0000000000000000 R11: ffffa22202dff200 R12: ffff8b3c3b76b000
  [570928.663059] R13: ffff8b3f2c0b0000 R14: ffff8b3dfe60d240 R15: ffff8b3c3b76b098
  [570928.663082] FS:  0000000000000000(0000) GS:ffff8b3f33d00000(0000) knlGS:0000000000000000
  [570928.663107] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [570928.663126] CR2: 00007f2a5bddc500 CR3: 00000001e1a10005 CR4: 00000000003706e0
  [570928.663184] Call Trace:
  [570928.663204]  css_release_work_fn+0x3f/0x240
  [570928.663254]  process_one_work+0x1a7/0x360
  [570928.663276]  worker_thread+0x30/0x390
  [570928.663291]  ? create_worker+0x1a0/0x1a0
  [570928.663305]  kthread+0x116/0x130
  [570928.663326]  ? kthread_flush_work_fn+0x10/0x10
  [570928.663344]  ret_from_fork+0x35/0x40
  

• ログの別のパターン:

  [1202369.537819] list_del corruption. next->prev should be ffff94c97f3f2098, but was ffff94c49aa94ad8
  [1202369.538099] ------------[ cut here ]------------
  [1202369.538221] kernel BUG at lib/list_debug.c:56!
  [1202369.538418] invalid opcode: 0000 [#1] SMP PTI
  [1202369.538539] CPU: 5 PID: 883812 Comm: kworker/5:1 Kdump: loaded Not tainted 4.18.0-305.3.1.el8_4.x86_64 #1
  [1202369.538689] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
  [1202369.538873] Workqueue: 0x0 (cgroup_destroy)
  [1202369.539025] RIP: 0010:__list_del_entry_valid.cold.1+0x20/0x4c
  [1202369.539178] Code: 65 10 93 e8 dc a4 cb ff 0f 0b 48 89 fe 48 89 c2 48 c7 c7 18 66 10 93 e8 c8 a4 cb ff 0f 0b 48 c7 c7 c8 66 10 93 e8 ba a4 cb ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 88 66 10 93 e8 a6 a4 cb ff 0f 0b
  [1202369.539488] RSP: 0018:ffffb32bd004fe60 EFLAGS: 00010046
  [1202369.539622] RAX: 0000000000000054 RBX: ffff94c97f3f2098 RCX: 0000000000000000
  [1202369.539769] RDX: 0000000000000000 RSI: ffff94c9a5f567c8 RDI: ffff94c9a5f567c8
  [1202369.539912] RBP: ffff94c97f3f2090 R08: 00000000000006d8 R09: 0000000000aaaaaa
  [1202369.540052] R10: 0000000000000000 R11: ffffb32bc4f40200 R12: ffff94c4ca311090
  [1202369.540195] R13: ffff94c9a5f697e0 R14: ffffffff920fea30 R15: 0000000000000000
  [1202369.540347] FS: 0000000000000000(0000) GS:ffff94c9a5f40000(0000) knlGS:0000000000000000
  [1202369.540493] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [1202369.540635] CR2: 00000000000000b0 CR3: 0000000408810003 CR4: 00000000001706e0
  [1202369.540805] Call Trace:
  [1202369.540942] move_linked_works+0x49/0xa0
  [1202369.541071] ? create_worker+0x1a0/0x1a0
  [1202369.541190] pwq_activate_delayed_work+0x3e/0xb0
  [1202369.541327] pwq_dec_nr_in_flight+0x5d/0x90
  [1202369.541459] worker_thread+0x30/0x390
  [1202369.541573] ? create_worker+0x1a0/0x1a0
  [1202369.541724] kthread+0x116/0x130
  [1202369.541849] ? kthread_flush_work_fn+0x10/0x10
  [1202369.541995] ret_from_fork+0x35/0x40