The kernel-rt crashed with a blocked task message where a sysctl knob hung_task_panic was enabled.
Issue
-
We were unable to SSH to a KVM guest.
-
The kernel-rt on the guest crashed with a blocked task message where a sysctl knob hung_task_panic was enabled.
[73993.802135] Kernel panic - not syncing: hung_task: blocked tasks
[73993.802138] CPU: 40 PID: 745 Comm: khungtaskd Kdump: loaded Tainted: G OE --------- - - 4.18.0-193.14.3.rt13.67.el8_2.x86_64 #1
[73993.802138] Hardware name: Quanta Cloud Technology Inc. QuantaGrid D52BE-2U/S5BE-MB 3UPI (LBG-1G), BIOS 3B13.RTN03 10/14/2019
[73993.802139] Call Trace:
[73993.802142] dump_stack+0x5c/0x80
[73993.802148] panic+0xe7/0x2a9
[73993.802151] watchdog+0x234/0x340
[73993.802153] ? hungtask_pm_notify+0x40/0x40
[73993.802154] kthread+0x112/0x130
[73993.802157] ? kthread_flush_work_fn+0x10/0x10
[73993.802159] ret_from_fork+0x1f/0x40
- Lots of blocked task messages were observed in kernel ring buffer before the crash.
[73993.777388] INFO: task ansible:652536 blocked for more than 600 seconds.
[73993.784084] Tainted: G OE --------- - - 4.18.0-193.14.3.rt13.67.el8_2.x86_64 #1
[73993.793120] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[73993.800946] ansible D 0 652536 77450 0x000803a0
[73993.800949] Call Trace:
[73993.800956] ? __schedule+0x316/0x7c0
[73993.800960] ? terminate_walk+0xdf/0x100
[73993.800963] schedule+0x39/0xd0
[73993.800966] __rt_mutex_slowlock+0xbe/0x130
[73993.800971] ? kmem_cache_alloc+0xb1/0x1d0
[73993.800974] rt_mutex_slowlock_locked+0xbc/0x270
[73993.800977] rt_mutex_slowlock+0x6d/0xc0
[73993.800982] sock_do_ioctl+0xe2/0x140
[73993.800987] ? seccomp_run_filters+0xa8/0x1a0
[73993.800993] ? selinux_file_alloc_security+0x32/0x50
[73993.800995] sock_ioctl+0x1a8/0x300
[73993.800998] ? selinux_file_ioctl+0x16f/0x210
[73993.801002] do_vfs_ioctl+0xa4/0x630
[73993.801006] ksys_ioctl+0x60/0x90
[73993.801009] __x64_sys_ioctl+0x16/0x20
[73993.801012] do_syscall_64+0x87/0x1a0
[73993.801015] entry_SYSCALL_64_after_hwframe+0x65/0xca
[73993.801017] RIP: 0033:0x7f1f654b287b
[73993.801021] Code: Bad RIP value.
[73993.801022] RSP: 002b:00007ffca78c0388 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[73993.801024] RAX: ffffffffffffffda RBX: 00000000ffffffff RCX: 00007f1f654b287b
[73993.801025] RDX: 00007ffca78c0390 RSI: 0000000000008912 RDI: 0000000000000003
[73993.801026] RBP: 0000000000000003 R08: 0000000000000000 R09: 0000557134682d08
[73993.801028] R10: 00007f1f66d770a0 R11: 0000000000000246 R12: 00007f1f603c6230
[73993.801029] R13: 00007ffca78c09e0 R14: 0000000000000000 R15: 00007ffca78c0b28
Environment
- kernel-4.18.0-193.14.3.rt13.67.el8_2
- KVM guest
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.