The kernel crashes due to the blocked tasks waiting for kthreadd to create workers. kthreadd that was responsible for creating workers was busy isolating the pages for the zone compaction.

Issue

• The kernel crashes due to the blocked tasks waiting for kthreadd to create workers.

[1140818.458383] INFO: task kworker/2:0:70006 blocked for more than 120 seconds.
[1140818.458453] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1140818.458511] kworker/2:0     D ffff8ebf56c05780     0 70006      2 0x00000080
[1140818.458523] Call Trace:
[1140818.458534]  [<ffffffffb2f89199>] schedule+0x29/0x70
[1140818.458538]  [<ffffffffb2f86e61>] schedule_timeout+0x221/0x2d0
[1140818.458544]  [<ffffffffb28d7402>] ? check_preempt_curr+0x92/0xa0
[1140818.458546]  [<ffffffffb28d7429>] ? ttwu_do_wakeup+0x19/0xe0
[1140818.458548]  [<ffffffffb2f8954d>] wait_for_completion+0xfd/0x140
[1140818.458551]  [<ffffffffb28dadd0>] ? wake_up_state+0x20/0x20
[1140818.458556]  [<ffffffffb28c5c9a>] kthread_create_on_node+0xaa/0x140
[1140818.458562]  [<ffffffffb28bee60>] ? manage_workers.isra.26+0x2a0/0x2a0
[1140818.458564]  [<ffffffffb28bea2b>] create_worker+0xeb/0x200
[1140818.458566]  [<ffffffffb28becb6>] manage_workers.isra.26+0xf6/0x2a0
[1140818.458568]  [<ffffffffb28bf1e3>] worker_thread+0x383/0x3c0
[1140818.458571]  [<ffffffffb28bee60>] ? manage_workers.isra.26+0x2a0/0x2a0
[1140818.458572]  [<ffffffffb28c5e41>] kthread+0xd1/0xe0
[1140818.458575]  [<ffffffffb28c5d70>] ? insert_kthread_work+0x40/0x40
[1140818.458580]  [<ffffffffb2f95ddd>] ret_from_fork_nospec_begin+0x7/0x21
[1140818.458582]  [<ffffffffb28c5d70>] ? insert_kthread_work+0x40/0x40
    ...

• At that time kthreadd that was responsible for creating workers was busy isolating the pages for the zone compaction.

[1140818.459880] CPU: 5 PID: 813 Comm: systemd-udevd Kdump: loaded Tainted: P           OE  ------------   3.10.0-1160.36.2.el7.x86_64 #1
[1140818.459881] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/19/2018
[1140818.459882] task: ffff8ed030cb2100 ti: ffff8ed030e68000 task.ti: ffff8ed030e68000
[1140818.459883] RIP: 0010:[<ffffffffb29e6e63>]  [<ffffffffb29e6e63>] isolate_freepages_block+0x83/0x320
[1140818.459884] RSP: 0018:ffff8ed030e6b9a8  EFLAGS: 00000206
[1140818.459886] RAX: 00000000ffffffff RBX: 0000000000cfd9d1 RCX: ffff8ed030e6bbf0
[1140818.459887] RDX: 0000000000000000 RSI: 0000000000000202 RDI: ffff8ed07ffda068
[1140818.459887] RBP: ffff8ed030e6ba40 R08: ffff8ed030e6bc08 R09: 0000000000cfda00
[1140818.459888] R10: ffff8ed07ffda000 R11: ffff8ed030e6bfd8 R12: ffffc9a3f3f60000
[1140818.459889] R13: ffffc9a3f3f67440 R14: 0000000000000000 R15: 00000000000001d1
[1140818.459890] FS:  00007fb3be3348c0(0000) GS:ffff8ed035740000(0000) knlGS:0000000000000000
[1140818.459891] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1140818.459892] CR2: 00007f1bcb8a5000 CR3: 0000001120442000 CR4: 00000000003607e0
[1140818.459893] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1140818.459893] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[1140818.459894] Call Trace:
[1140818.459895]  [<ffffffffb29e7468>] compaction_alloc+0x248/0x360
[1140818.459896]  [<ffffffffb2a305ac>] migrate_pages+0x21c/0x7f0
[1140818.459897]  [<ffffffffb29e7220>] ? __reset_isolation_suitable+0x120/0x120
[1140818.459904]  [<ffffffffb29e838d>] compact_zone+0x34d/0x4d0
[1140818.459904]  [<ffffffffb29e857d>] compact_zone_order+0x6d/0xa0
[1140818.459905]  [<ffffffffb29e8921>] try_to_compact_pages+0x121/0x1a0
[1140818.459906]  [<ffffffffb2f7ea53>] __alloc_pages_direct_compact+0xac/0x193
[1140818.459907]  [<ffffffffb29c951c>] __alloc_pages_nodemask+0x7bc/0xbe0
[1140818.459908]  [<ffffffffb289880d>] copy_process+0x1dd/0x1a80
[1140818.459909]  [<ffffffffb289a261>] do_fork+0x91/0x330
[1140818.459910]  [<ffffffffb289a586>] SyS_clone+0x16/0x20
[1140818.459911]  [<ffffffffb2f96374>] stub_clone+0x44/0x70
[1140818.459911]  [<ffffffffb2f95f92>] ? system_call_fastpath+0x25/0x2a
[1140818.459913] Code: 4f 58 48 83 c0 40 45 89 c6 45 31 e4 c7 45 b4 00 00 00 00 45 31 ff 48 89 4d a0 48 89 45 98 f6 c3 1f 0f 84 b1 00 00 00 41 8b 45 18 <41> 83 c7 01 4d 85 e4 4d 0f 44 e5 83 f8 80 0f 84 d1 00 00 00 45 
    ...

• And then finally the kernel crashed due to the blocked tasks.

[1140818.459915] Kernel panic - not syncing: hung_task: blocked tasks
[1140818.459981] CPU: 0 PID: 40 Comm: khungtaskd Kdump: loaded Tainted: P           OE  ------------   3.10.0-1160.36.2.el7.x86_64 #1
[1140818.460114] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/19/2018
[1140818.460159] Call Trace:
[1140818.460205]  [<ffffffffb2f83559>] dump_stack+0x19/0x1b
[1140818.460254]  [<ffffffffb2f7d261>] panic+0xe8/0x21f
[1140818.460299]  [<ffffffffb294e93e>] watchdog+0x26e/0x2c0
[1140818.460358]  [<ffffffffb294e6d0>] ? reset_hung_task_detector+0x20/0x20
[1140818.460419]  [<ffffffffb28c5e41>] kthread+0xd1/0xe0
[1140818.460475]  [<ffffffffb28c5d70>] ? insert_kthread_work+0x40/0x40
[1140818.460500]  [<ffffffffb2f95ddd>] ret_from_fork_nospec_begin+0x7/0x21
[1140818.471874]  [<ffffffffb28c5d70>] ? insert_kthread_work+0x40/0x40