system hangs under load due to blocked tasks and high cpu load
Issue
Upon starting the application on a system with storage foundation installed the server may hang or show high load average and CPU activity. Many of the blocked processes are ps, pgrep, and other simple commands. The blocked tasks stacks indicate most are sleeping on __down_read or __down_write i.e. access to a semaphore. Example stack traces below. From the crash analysis we see that all blocked tasks (+8000) are sleeping on the mm_struct.mmap_sem for this task's thread group. This is critical as it differentiates other issues with similar symptoms from this.
crash> bt ffff812fff92a0c0
PID: 17770 TASK: ffff812fff92a0c0 CPU: 16 COMMAND: "pmdtm"
#0 [ffff812fa8de7e38] schedule at ffffffff80062fa0
#1 [ffff812fa8de7f10] __down_write_nested at ffffffff800645f3
#2 [ffff812fa8de7f50] sys_brk at ffffffff80017488
#3 [ffff812fa8de7f80] tracesys at ffffffff8005d29e (via system_call)
RIP: 00000033038cd6da RSP: 00002ace183e4c38 RFLAGS: 00000287
RAX: ffffffffffffffda RBX: ffffffff8005d29e RCX: ffffffffffffffff
RDX: 0000000000021720 RSI: 000000000eb58000 RDI: 000000000eb58000
RBP: 0000000000000001 R8: 0000000000000004 R9: 0000000000000003
R10: 005200450056005f R11: 0000000000000287 R12: 0000003303b549e0
R13: 0000000000022000 R14: 0000000000022000 R15: 000000000eb36000
ORIG_RAX: 000000000000000c CS: 0033 SS: 002b
crash> bt ffff816024ecb080
PID: 23013 TASK: ffff816024ecb080 CPU: 10 COMMAND: "perfd"
#0 [ffff81601999bcc8] schedule at ffffffff80062fa0
#1 [ffff81601999bda0] __down_read at ffffffff8006468c
#2 [ffff81601999bde0] __access_process_vm at ffffffff800c74e0
#3 [ffff81601999be50] access_process_vm at ffffffff800c765b
#4 [ffff81601999be90] proc_pid_cmdline at ffffffff8010d970
#5 [ffff81601999bed0] proc_info_read at ffffffff8010e0b6
#6 [ffff81601999bf10] vfs_read at ffffffff8000b732
#7 [ffff81601999bf40] sys_read at ffffffff80011db5
#8 [ffff81601999bf80] tracesys at ffffffff8005d29e (via system_call)
RIP: 00000033038c71fb RSP: 00007fffab53af40 RFLAGS: 00000202
RAX: ffffffffffffffda RBX: ffffffff8005d29e RCX: ffffffffffffffff
RDX: 0000000000000fff RSI: 00007fffab53af90 RDI: 0000000000000008
RBP: 0000000000000000 R8: 00002ba578aadf80 R9: 000000330391ad60
R10: 0000000000000008 R11: 0000000000000202 R12: 0000000000000008
R13: 0000000000000000 R14: 00007fffab53bf90 R15: 000000017898101c
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
Environment
o Red Hat Enterprise Linux ( All Version )
o Veritas Storage Foundation
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
