The server is getting highly loaded and unresponsive. Lots of tasks are stuck waiting on the rw_semaphore being held by the task that is running the functions from the 3rd party module [vxfs]

Issue

• The server is getting highly loaded and unresponsive. Lots of tasks are stuck waiting on the rw_semaphore being held by the task that is running the functions from the 3rd party module [vxfs]

crash> foreach UN bt|awk '/#2 / || /#3 / {print $3,$5}'|sort|uniq -c
    168 call_rwsem_down_read_failed ffffffffa8597568
     25 call_rwsem_down_write_failed ffffffffa8597597
      7 mutex_lock ffffffffa898548f
      7 __mutex_lock_slowpath ffffffffa89860b7
    168 rwsem_down_read_failed ffffffffa8988c15
     25 rwsem_down_write_failed ffffffffa8988965
    628 schedule_timeout ffffffffa8984c58
    628 vx_delay ffffffffc07612fe     <=-----  [ total 628 tasks blocked on vx_delay() ]
      4 vx_event_wait ffffffffc07141ef
      3 vxg_grant_sleep ffffffffc33b8c97
     55 vxg_ilock_wait ffffffffc33b80b5
     58 vxg_svar_sleep_unlock ffffffffc339f159
      4 vx_svar_sleep_unlock ffffffffc0713a00

• The semaphore is taken by the below process running vxfs functions:

crash> bt
PID: 71321  TASK: ffff928a0ad62100  CPU: 16  COMMAND: "pmdtm"
 #0 [ffff921190e5b4c8] __schedule at ffffffffa8986d07
 #1 [ffff921190e5b558] schedule at ffffffffa8987229
 #2 [ffff921190e5b568] schedule_timeout at ffffffffa8984c58
 #3 [ffff921190e5b610] vx_delay at ffffffffc07612fe [vxfs]
 #4 [ffff921190e5b620] vx_traninit at ffffffffc07d3338 [vxfs]
 #5 [ffff921190e5b670] vx_write_alloc3 at ffffffffc0614be9 [vxfs]
 #6 [ffff921190e5b7f8] vx_tran_write_alloc at ffffffffc0615919 [vxfs]
 #7 [ffff921190e5b868] vx_cfs_prealloc at ffffffffc05fa2c0 [vxfs]
 #8 [ffff921190e5b938] vx_write_alloc2 at ffffffffc0614aeb [vxfs]
 #9 [ffff921190e5b988] vx_write_alloc at ffffffffc0804398 [vxfs]
#10 [ffff921190e5ba78] vx_write1 at ffffffffc0805036 [vxfs]
#11 [ffff921190e5bbe0] vx_write_common_slow at ffffffffc0806519 [vxfs]
#12 [ffff921190e5bcc0] vx_write_common at ffffffffc0807864 [vxfs]
#13 [ffff921190e5bdd8] vx_write at ffffffffc07721e7 [vxfs]
#14 [ffff921190e5bec8] vfs_write at ffffffffa844db90
#15 [ffff921190e5bf08] sys_write at ffffffffa844e96f
#16 [ffff921190e5bf50] system_call_fastpath at ffffffffa8994f92
    RIP: 00007f0f4db056fd  RSP: 00007f0e4f1e87d8  RFLAGS: 00000203
    RAX: 0000000000000001  RBX: 0000000000002000  RCX: 0000000000000000
    RDX: 0000000000002000  RSI: 00007f0ea7867010  RDI: 0000000000000168
    RBP: 00007f0ea7867010   R8: 00007f0e4f1e87ac   R9: 0000000000000000
    R10: 0000000000000000  R11: 0000000000000293  R12: 0000000000002000
    R13: 0000000000000168  R14: 0000000000002000  R15: 0000000000000000
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b