The kernel-rt crashed with a blocked task message where a sysctl knob hung_task_panic was enabled.

Solution Unverified - Updated -

Issue

  • We were unable to SSH to a KVM guest.

  • The kernel-rt on the guest crashed with a blocked task message where a sysctl knob hung_task_panic was enabled.

[73993.802135] Kernel panic - not syncing: hung_task: blocked tasks
[73993.802138] CPU: 40 PID: 745 Comm: khungtaskd Kdump: loaded Tainted: G           OE    --------- -  - 4.18.0-193.14.3.rt13.67.el8_2.x86_64 #1
[73993.802138] Hardware name: Quanta Cloud Technology Inc. QuantaGrid D52BE-2U/S5BE-MB 3UPI (LBG-1G), BIOS 3B13.RTN03 10/14/2019
[73993.802139] Call Trace:
[73993.802142]  dump_stack+0x5c/0x80
[73993.802148]  panic+0xe7/0x2a9
[73993.802151]  watchdog+0x234/0x340
[73993.802153]  ? hungtask_pm_notify+0x40/0x40
[73993.802154]  kthread+0x112/0x130
[73993.802157]  ? kthread_flush_work_fn+0x10/0x10
[73993.802159]  ret_from_fork+0x1f/0x40
[73993.777388] INFO: task ansible:652536 blocked for more than 600 seconds.
[73993.784084]       Tainted: G           OE    --------- -  - 4.18.0-193.14.3.rt13.67.el8_2.x86_64 #1
[73993.793120] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[73993.800946] ansible         D    0 652536  77450 0x000803a0
[73993.800949] Call Trace:
[73993.800956]  ? __schedule+0x316/0x7c0
[73993.800960]  ? terminate_walk+0xdf/0x100
[73993.800963]  schedule+0x39/0xd0
[73993.800966]  __rt_mutex_slowlock+0xbe/0x130
[73993.800971]  ? kmem_cache_alloc+0xb1/0x1d0
[73993.800974]  rt_mutex_slowlock_locked+0xbc/0x270
[73993.800977]  rt_mutex_slowlock+0x6d/0xc0
[73993.800982]  sock_do_ioctl+0xe2/0x140
[73993.800987]  ? seccomp_run_filters+0xa8/0x1a0
[73993.800993]  ? selinux_file_alloc_security+0x32/0x50
[73993.800995]  sock_ioctl+0x1a8/0x300
[73993.800998]  ? selinux_file_ioctl+0x16f/0x210
[73993.801002]  do_vfs_ioctl+0xa4/0x630
[73993.801006]  ksys_ioctl+0x60/0x90
[73993.801009]  __x64_sys_ioctl+0x16/0x20
[73993.801012]  do_syscall_64+0x87/0x1a0
[73993.801015]  entry_SYSCALL_64_after_hwframe+0x65/0xca
[73993.801017] RIP: 0033:0x7f1f654b287b
[73993.801021] Code: Bad RIP value.
[73993.801022] RSP: 002b:00007ffca78c0388 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[73993.801024] RAX: ffffffffffffffda RBX: 00000000ffffffff RCX: 00007f1f654b287b
[73993.801025] RDX: 00007ffca78c0390 RSI: 0000000000008912 RDI: 0000000000000003
[73993.801026] RBP: 0000000000000003 R08: 0000000000000000 R09: 0000557134682d08
[73993.801028] R10: 00007f1f66d770a0 R11: 0000000000000246 R12: 00007f1f603c6230
[73993.801029] R13: 00007ffca78c09e0 R14: 0000000000000000 R15: 00007ffca78c0b28

Environment

  • kernel-4.18.0-193.14.3.rt13.67.el8_2
  • KVM guest

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In