The kernel is getting hung up where lots of tasks are getting stuck waiting on the buffer_head that is being locked by another task running prune_super().
Issue
- The kernel is getting hung up where lots of tasks are getting stuck waiting on the buffer_head that is being locked by another task running prune_super() for reclaiming inodes for page reclaiming.
[ 6962.278739] INFO: task jbd2/loop8-8:17606 blocked for more than 120 seconds.
[ 6962.278801] INFO: task jsivm:17794 blocked for more than 120 seconds.
[ 6962.278823] INFO: task jsivm:17809 blocked for more than 120 seconds.
[ 6962.278841] INFO: task jsivm:17810 blocked for more than 120 seconds.
[ 6962.278936] INFO: task jsivm:17811 blocked for more than 120 seconds.
[ 6962.279009] INFO: task jsivm:17987 blocked for more than 120 seconds.
[ 6962.279023] INFO: task jsivm:18019 blocked for more than 120 seconds.
[ 6962.279068] INFO: task ps:29385 blocked for more than 120 seconds.
[ 6962.279091] INFO: task ps:29468 blocked for more than 120 seconds.
[ 7082.277489] INFO: task jbd2/loop8-8:17606 blocked for more than 120 seconds.
PID: 17811 TASK: ffff9f84d6136300 CPU: 4 COMMAND: "appthread"
#0 [ffff9f6f88387878] __schedule at ffffffffa4f8899f
#1 [ffff9f6f88387908] schedule at ffffffffa4f88eb9
#2 [ffff9f6f88387918] schedule_timeout at ffffffffa4f86991
#3 [ffff9f6f883879c0] io_schedule_timeout at ffffffffa4f8857d
#4 [ffff9f6f883879f0] io_schedule at ffffffffa4f88618
#5 [ffff9f6f88387a00] bit_wait_io at ffffffffa4f86fe1
#6 [ffff9f6f88387a18] __wait_on_bit_lock at ffffffffa4f86b91
#7 [ffff9f6f88387a58] out_of_line_wait_on_bit_lock at ffffffffa4f86de1
#8 [ffff9f6f88387ad0] __lock_buffer at ffffffffa4a86d52
#9 [ffff9f6f88387ae0] __ext4_get_inode_loc at ffffffffc04b643a [ext4]
...
PID: 17813 TASK: ffff9f8c438f9080 CPU: 3 COMMAND: "appthread"
#0 [ffff9f8e3f2c8e48] crash_nmi_callback at ffffffffa4858567
#1 [ffff9f8e3f2c8e58] nmi_handle at ffffffffa4f8d93c
#2 [ffff9f8e3f2c8eb0] do_nmi at ffffffffa4f8db5d
#3 [ffff9f8e3f2c8ef0] end_repeat_nmi at ffffffffa4f8cd9c
[exception RIP: prune_super+9]
RIP: ffffffffa4a50869 RSP: ffff9f70347a3868 RFLAGS: 00200206
RAX: ffffffffa4a50860 RBX: 0000000000000400 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff9f70347a3998 RDI: ffff9f6fc5fe4bb0
RBP: ffff9f70347a3868 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000000 R11: fff00000f0000000 R12: 000000000000001d
R13: 0000000000000000 R14: ffff9f6fc5fe4bb0 R15: ffff9f70347a3998
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000
--- <NMI exception stack> ---
#4 [ffff9f70347a3868] prune_super at ffffffffa4a50869
...
#6 [ffff9f70347a3918] do_try_to_free_pages at ffffffffa49d4d8a
#7 [ffff9f70347a3990] try_to_free_pages at ffffffffa49d4fdc
#8 [ffff9f70347a3a28] free_more_memory at ffffffffa4a85d5e
#9 [ffff9f70347a3a60] __getblk at ffffffffa4a870cb
#10 [ffff9f70347a3ac0] __breadahead at ffffffffa4a88b71
#11 [ffff9f70347a3ae0] __ext4_get_inode_loc at ffffffffc04b63fa [ext4]
#12 [ffff9f70347a3b48] ext4_get_inode_loc at ffffffffc04b894d [ext4]
...
Environment
- Red Hat Enterprise Linux 7.9.z kernel-3.10.0-1160.24.1.el7
- An instance of a RHEL virtual machine hosted on Google cloud infrastructure.
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.