Severe contention on kernfs_rwsem that often happens on RHCOS shipped with RHOCP4.9-4.11 causes a system hang

Solution Unverified - Updated -

Issue

  • Severe contention on kernfs_rwsem that often happens on RHCOS shipped with RHOCP4.9-4.11 causes a system hang
 #0 [ffffbeb85a387b28] __schedule at ffffffff977d5581
 #1 [ffffbeb85a387b80] schedule at ffffffff977d5b45
 #2 [ffffbeb85a387b90] schedule_preempt_disabled at ffffffff977d5e6a
 #3 [ffffbeb85a387b98] rwsem_down_read_slowpath at ffffffff977d848e
 #4 [ffffbeb85a387c38] down_read at ffffffff977d86a5
 #5 [ffffbeb85a387c48] kernfs_iop_permission at ffffffff971e9809
 #6 [ffffbeb85a387c70] inode_permission at ffffffff9715fdfa
 #7 [ffffbeb85a387c98] link_path_walk at ffffffff971604eb
 #8 [ffffbeb85a387cf8] path_lookupat at ffffffff97160767
 #9 [ffffbeb85a387d58] filename_lookup at ffffffff97164c40
#10 [ffffbeb85a387e68] user_statfs at ffffffff9718fc43
#11 [ffffbeb85a387ea8] __do_sys_statfs at ffffffff9718fcc0
#12 [ffffbeb85a387f38] do_syscall_64 at ffffffff96e043ab
#13 [ffffbeb85a387f50] entry_SYSCALL_64_after_hwframe at ffffffff978000a9
    RIP: 00007f4f963a117b  RSP: 00007ffec2fc3268  RFLAGS: 00000246
    RAX: ffffffffffffffda  RBX: 00007f4f96d63f53  RCX: 00007f4f963a117b
    RDX: 00007f4f964eb6f4  RSI: 00007ffec2fc32e0  RDI: 00007f4f96d63f53
    RBP: 00007ffec2fc32e0   R8: 00007f4f975be9c0   R9: 000000000000002f
    R10: fffffffffffffe98  R11: 0000000000000246  R12: 00007ffec2fc3468
    R13: 00007ffec2fc3490  R14: 00007f4f96f6a690  R15: 0000000000000000
    ORIG_RAX: 0000000000000089  CS: 0033  SS: 002b

 #0 [ffffbeb85a457aa8] __schedule at ffffffff977d5581
 #1 [ffffbeb85a457b00] schedule at ffffffff977d5b45
 #2 [ffffbeb85a457b10] schedule_preempt_disabled at ffffffff977d5e6a
 #3 [ffffbeb85a457b18] rwsem_down_read_slowpath at ffffffff977d848e
 #4 [ffffbeb85a457bb0] down_read at ffffffff977d86a5
 #5 [ffffbeb85a457bc0] kernfs_dop_revalidate at ffffffff971eacb5
 #6 [ffffbeb85a457be0] lookup_fast at ffffffff9715ee33
 #7 [ffffbeb85a457c38] walk_component at ffffffff9715f788
 #8 [ffffbeb85a457c98] link_path_walk at ffffffff971603ab
 #9 [ffffbeb85a457cf8] path_lookupat at ffffffff97160767
#10 [ffffbeb85a457d58] filename_lookup at ffffffff97164c40
#11 [ffffbeb85a457e68] user_statfs at ffffffff9718fc43
#12 [ffffbeb85a457ea8] __do_sys_statfs at ffffffff9718fcc0
#13 [ffffbeb85a457f38] do_syscall_64 at ffffffff96e043ab
#14 [ffffbeb85a457f50] entry_SYSCALL_64_after_hwframe at ffffffff978000a9
    RIP: 000055e53bbda21b  RSP: 000000c00014d898  RFLAGS: 00000216
    RAX: ffffffffffffffda  RBX: 000000c000030500  RCX: 000055e53bbda21b
    RDX: 0000000000000000  RSI: 000000c00014d948  RDI: 000000c00013c030
    RBP: 000000c00014d8f8   R8: 00007f55360b7501   R9: 0000000000000001
    R10: 00007f5535f37008  R11: 0000000000000216  R12: 0000000000000000
    R13: 0000000000000000  R14: 000000c0000021a0  R15: ffffffffffffffff
    ORIG_RAX: 0000000000000089  CS: 0033  SS: 002b

Environment

  • Red Hat OpenShift Container Platform 4
  • Red Hat Enterprise Linux CoreOS shipped with RHOCP 4
  • Kubelet threads
  • Red Hat Enterprise Linux 8

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content