The kernel-rt crashed with a blocked task message where a sysctl knob hung_task_panic was enabled.

Solution Unverified - Updated -

Issue

  • We were unable to SSH to a KVM guest.

  • The kernel-rt on the guest crashed with a blocked task message where a sysctl knob hung_task_panic was enabled.

[73993.802135] Kernel panic - not syncing: hung_task: blocked tasks
[73993.802138] CPU: 40 PID: 745 Comm: khungtaskd Kdump: loaded Tainted: G           OE    --------- -  - 4.18.0-193.14.3.rt13.67.el8_2.x86_64 #1
[73993.802138] Hardware name: Quanta Cloud Technology Inc. QuantaGrid D52BE-2U/S5BE-MB 3UPI (LBG-1G), BIOS 3B13.RTN03 10/14/2019
[73993.802139] Call Trace:
[73993.802142]  dump_stack+0x5c/0x80
[73993.802148]  panic+0xe7/0x2a9
[73993.802151]  watchdog+0x234/0x340
[73993.802153]  ? hungtask_pm_notify+0x40/0x40
[73993.802154]  kthread+0x112/0x130
[73993.802157]  ? kthread_flush_work_fn+0x10/0x10
[73993.802159]  ret_from_fork+0x1f/0x40
[73993.777388] INFO: task ansible:652536 blocked for more than 600 seconds.
[73993.784084]       Tainted: G           OE    --------- -  - 4.18.0-193.14.3.rt13.67.el8_2.x86_64 #1
[73993.793120] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[73993.800946] ansible         D    0 652536  77450 0x000803a0
[73993.800949] Call Trace:
[73993.800956]  ? __schedule+0x316/0x7c0
[73993.800960]  ? terminate_walk+0xdf/0x100
[73993.800963]  schedule+0x39/0xd0
[73993.800966]  __rt_mutex_slowlock+0xbe/0x130
[73993.800971]  ? kmem_cache_alloc+0xb1/0x1d0
[73993.800974]  rt_mutex_slowlock_locked+0xbc/0x270
[73993.800977]  rt_mutex_slowlock+0x6d/0xc0
[73993.800982]  sock_do_ioctl+0xe2/0x140
[73993.800987]  ? seccomp_run_filters+0xa8/0x1a0
[73993.800993]  ? selinux_file_alloc_security+0x32/0x50
[73993.800995]  sock_ioctl+0x1a8/0x300
[73993.800998]  ? selinux_file_ioctl+0x16f/0x210
[73993.801002]  do_vfs_ioctl+0xa4/0x630
[73993.801006]  ksys_ioctl+0x60/0x90
[73993.801009]  __x64_sys_ioctl+0x16/0x20
[73993.801012]  do_syscall_64+0x87/0x1a0
[73993.801015]  entry_SYSCALL_64_after_hwframe+0x65/0xca
[73993.801017] RIP: 0033:0x7f1f654b287b
[73993.801021] Code: Bad RIP value.
[73993.801022] RSP: 002b:00007ffca78c0388 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[73993.801024] RAX: ffffffffffffffda RBX: 00000000ffffffff RCX: 00007f1f654b287b
[73993.801025] RDX: 00007ffca78c0390 RSI: 0000000000008912 RDI: 0000000000000003
[73993.801026] RBP: 0000000000000003 R08: 0000000000000000 R09: 0000557134682d08
[73993.801028] R10: 00007f1f66d770a0 R11: 0000000000000246 R12: 00007f1f603c6230
[73993.801029] R13: 00007ffca78c09e0 R14: 0000000000000000 R15: 00007ffca78c0b28

Environment

  • kernel-4.18.0-193.14.3.rt13.67.el8_2
  • KVM guest

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content