System hang due to large number of blocked tasks - module CyProtectDrv in context
Issue
- Server hang due to large number of blocked tasks and a reboot was required
- High load average
sys | grep LOAD
LOAD AVERAGE: 4837.11, 4837.12, 4837.09
Oldest task:
crash> bt 15868
PID: 15868 TASK: ffff881053b6f520 CPU: 20 COMMAND: "awk"
#0 [ffff880456caf860] schedule at ffffffff81552d9a
#1 [ffff880456caf948] rwsem_down_failed_common at ffffffff81556065
#2 [ffff880456caf9a8] rwsem_down_read_failed at ffffffff815561f6
#3 [ffff880456caf9e8] call_rwsem_down_read_failed at ffffffff812ac864
#4 [ffff880456cafa50] __do_page_fault at ffffffff810533e4
#5 [ffff880456cafb70] do_page_fault at ffffffff8155a29e
#6 [ffff880456cafba0] page_fault at ffffffff81557285
[exception RIP: strncpy_from_user+91]
RIP: ffffffff812acd9b RSP: ffff880456cafc58 RFLAGS: 00010202
RAX: 0000000000000000 RBX: 000000000000000b RCX: 000000000000000b
RDX: 0000000000000000 RSI: 00007ffd86257fe4 RDI: ffff880456cafc98
RBP: ffff880456cafc78 R8: 000000000000004c R9: 00000000000000e6
R10: ffff880456caff58 R11: 0000000000000000 R12: 00007ffd86257fe4
R13: 000000000000000b R14: ffff880456cafd20 R15: 0000000000000001
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#7 [ffff880456cafc80] cy_strncpy_from_user at ffffffffa03e5bce [CyProtectDrv]
#8 [ffff880456cafc90] MemDef_GetDYLD_callback at ffffffffa03e7e46 [CyProtectDrv]
#9 [ffff880456cafce0] MemDef_ScanStack.clone.0 at ffffffffa03e8861 [CyProtectDrv]
#10 [ffff880456cafd70] MemDef_OnProcInited at ffffffffa03e92b2 [CyProtectDrv]
#11 [ffff880456cafe00] cyl_load_binary at ffffffffa03ead7f [CyProtectDrv]
#12 [ffff880456cafe50] search_binary_handler at ffffffff811a589b
#13 [ffff880456cafec0] do_execve at ffffffff811a6647
#14 [ffff880456caff20] sys_execve at ffffffff810095ea
#15 [ffff880456caff50] stub_execve at ffffffff8155e979
RIP: 0000003823800b30 RSP: 00007ffd86256620 RFLAGS: 00000200
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: 000000000000003b CS: 0033 SS: 002b
WARNING: possibly bogus exception frame
crash>
Environment
- RHEL 6
- Third party vendor module - CyProtectDrv
seen in module version 2.0.1480.324
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.