The kernel is getting hung up where lots of tasks are getting stuck waiting on the buffer_head that is being locked by another task running prune_super().

Solution Unverified - Updated -

Issue

  • The kernel is getting hung up where lots of tasks are getting stuck waiting on the buffer_head that is being locked by another task running prune_super() for reclaiming inodes for page reclaiming.
[ 6962.278739] INFO: task jbd2/loop8-8:17606 blocked for more than 120 seconds.
[ 6962.278801] INFO: task jsivm:17794 blocked for more than 120 seconds.
[ 6962.278823] INFO: task jsivm:17809 blocked for more than 120 seconds.
[ 6962.278841] INFO: task jsivm:17810 blocked for more than 120 seconds.
[ 6962.278936] INFO: task jsivm:17811 blocked for more than 120 seconds.
[ 6962.279009] INFO: task jsivm:17987 blocked for more than 120 seconds.
[ 6962.279023] INFO: task jsivm:18019 blocked for more than 120 seconds.
[ 6962.279068] INFO: task ps:29385 blocked for more than 120 seconds.
[ 6962.279091] INFO: task ps:29468 blocked for more than 120 seconds.
[ 7082.277489] INFO: task jbd2/loop8-8:17606 blocked for more than 120 seconds.

PID: 17811  TASK: ffff9f84d6136300  CPU: 4   COMMAND: "appthread"
 #0 [ffff9f6f88387878] __schedule at ffffffffa4f8899f
 #1 [ffff9f6f88387908] schedule at ffffffffa4f88eb9
 #2 [ffff9f6f88387918] schedule_timeout at ffffffffa4f86991
 #3 [ffff9f6f883879c0] io_schedule_timeout at ffffffffa4f8857d
 #4 [ffff9f6f883879f0] io_schedule at ffffffffa4f88618
 #5 [ffff9f6f88387a00] bit_wait_io at ffffffffa4f86fe1
 #6 [ffff9f6f88387a18] __wait_on_bit_lock at ffffffffa4f86b91
 #7 [ffff9f6f88387a58] out_of_line_wait_on_bit_lock at ffffffffa4f86de1
 #8 [ffff9f6f88387ad0] __lock_buffer at ffffffffa4a86d52
 #9 [ffff9f6f88387ae0] __ext4_get_inode_loc at ffffffffc04b643a [ext4]
    ...

PID: 17813  TASK: ffff9f8c438f9080  CPU: 3   COMMAND: "appthread"
 #0 [ffff9f8e3f2c8e48] crash_nmi_callback at ffffffffa4858567
 #1 [ffff9f8e3f2c8e58] nmi_handle at ffffffffa4f8d93c
 #2 [ffff9f8e3f2c8eb0] do_nmi at ffffffffa4f8db5d
 #3 [ffff9f8e3f2c8ef0] end_repeat_nmi at ffffffffa4f8cd9c
    [exception RIP: prune_super+9]
    RIP: ffffffffa4a50869  RSP: ffff9f70347a3868  RFLAGS: 00200206
    RAX: ffffffffa4a50860  RBX: 0000000000000400  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: ffff9f70347a3998  RDI: ffff9f6fc5fe4bb0
    RBP: ffff9f70347a3868   R8: 0000000000000000   R9: 0000000000000000
    R10: 0000000000000000  R11: fff00000f0000000  R12: 000000000000001d
    R13: 0000000000000000  R14: ffff9f6fc5fe4bb0  R15: ffff9f70347a3998
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0000
--- <NMI exception stack> ---
 #4 [ffff9f70347a3868] prune_super at ffffffffa4a50869
    ...
 #6 [ffff9f70347a3918] do_try_to_free_pages at ffffffffa49d4d8a
 #7 [ffff9f70347a3990] try_to_free_pages at ffffffffa49d4fdc
 #8 [ffff9f70347a3a28] free_more_memory at ffffffffa4a85d5e
 #9 [ffff9f70347a3a60] __getblk at ffffffffa4a870cb
#10 [ffff9f70347a3ac0] __breadahead at ffffffffa4a88b71
#11 [ffff9f70347a3ae0] __ext4_get_inode_loc at ffffffffc04b63fa [ext4]
#12 [ffff9f70347a3b48] ext4_get_inode_loc at ffffffffc04b894d [ext4]
    ...

Environment

  • Red Hat Enterprise Linux 7.9.z kernel-3.10.0-1160.24.1.el7
  • An instance of a RHEL virtual machine hosted on Google cloud infrastructure.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content