System hang due to large number of blocked tasks - module CyProtectDrv in context

Solution Verified - Updated -

Issue

  • Server hang due to large number of blocked tasks and a reboot was required
  • High load average
sys | grep LOAD
LOAD AVERAGE: 4837.11, 4837.12, 4837.09

Oldest task:
crash> bt 15868 
PID: 15868  TASK: ffff881053b6f520  CPU: 20  COMMAND: "awk"
 #0 [ffff880456caf860] schedule at ffffffff81552d9a
 #1 [ffff880456caf948] rwsem_down_failed_common at ffffffff81556065
 #2 [ffff880456caf9a8] rwsem_down_read_failed at ffffffff815561f6
 #3 [ffff880456caf9e8] call_rwsem_down_read_failed at ffffffff812ac864
 #4 [ffff880456cafa50] __do_page_fault at ffffffff810533e4
 #5 [ffff880456cafb70] do_page_fault at ffffffff8155a29e
 #6 [ffff880456cafba0] page_fault at ffffffff81557285
    [exception RIP: strncpy_from_user+91]
    RIP: ffffffff812acd9b  RSP: ffff880456cafc58  RFLAGS: 00010202
    RAX: 0000000000000000  RBX: 000000000000000b  RCX: 000000000000000b
    RDX: 0000000000000000  RSI: 00007ffd86257fe4  RDI: ffff880456cafc98
    RBP: ffff880456cafc78   R8: 000000000000004c   R9: 00000000000000e6
    R10: ffff880456caff58  R11: 0000000000000000  R12: 00007ffd86257fe4
    R13: 000000000000000b  R14: ffff880456cafd20  R15: 0000000000000001
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffff880456cafc80] cy_strncpy_from_user at ffffffffa03e5bce [CyProtectDrv]
 #8 [ffff880456cafc90] MemDef_GetDYLD_callback at ffffffffa03e7e46 [CyProtectDrv]
 #9 [ffff880456cafce0] MemDef_ScanStack.clone.0 at ffffffffa03e8861 [CyProtectDrv]
#10 [ffff880456cafd70] MemDef_OnProcInited at ffffffffa03e92b2 [CyProtectDrv]
#11 [ffff880456cafe00] cyl_load_binary at ffffffffa03ead7f [CyProtectDrv]
#12 [ffff880456cafe50] search_binary_handler at ffffffff811a589b
#13 [ffff880456cafec0] do_execve at ffffffff811a6647
#14 [ffff880456caff20] sys_execve at ffffffff810095ea
#15 [ffff880456caff50] stub_execve at ffffffff8155e979
    RIP: 0000003823800b30  RSP: 00007ffd86256620  RFLAGS: 00000200
    RAX: 0000000000000000  RBX: 0000000000000000  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: 0000000000000000  RDI: 0000000000000000
    RBP: 0000000000000000   R8: 0000000000000000   R9: 0000000000000000
    R10: 0000000000000000  R11: 0000000000000000  R12: 0000000000000000
    R13: 0000000000000000  R14: 0000000000000000  R15: 0000000000000000
    ORIG_RAX: 000000000000003b  CS: 0033  SS: 002b
WARNING: possibly bogus exception frame
crash> 

Environment

  • RHEL 6
  • Third party vendor module - CyProtectDrv
    seen in module version 2.0.1480.324

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content