raid1d task deadlocked due to memory shortage
Issue
- System hung and the root cause is believed to be that the raid1d thread is deadlocked.
- It needs to write pages out to free up memory but can't allocate an I/O request because there's no free memory.
PID: 848 TASK: ffff81042fc8a860 CPU: 5 COMMAND: "md1_raid1"
#0 [ffff81042dead710] schedule at ffffffff80062fa0
#1 [ffff81042dead7e8] schedule_timeout at ffffffff8006389f
#2 [ffff81042dead838] io_schedule_timeout at ffffffff80063241
#3 [ffff81042dead868] blk_congestion_wait at ffffffff8003b359
#4 [ffff81042dead8b8] throttle_vm_writeout at ffffffff800535a9
#5 [ffff81042dead8d8] shrink_zone at ffffffff800132eb
#6 [ffff81042dead918] try_to_free_pages at ffffffff800ce8ba
#7 [ffff81042dead9a8] __alloc_pages at ffffffff8000f5f6
#8 [ffff81042deada18] cache_grow at ffffffff80017aa8
#9 [ffff81042deada68] cache_alloc_refill at ffffffff8005be11
#10 [ffff81042deadaa8] kmem_cache_alloc at ffffffff8000ad2e
#11 [ffff81042deadac8] mempool_alloc at ffffffff80023469
#12 [ffff81042deadb28] cfq_set_request at ffffffff8001caeb
#13 [ffff81042deadba8] get_request at ffffffff8001361c
#14 [ffff81042deadbf8] get_request_wait at ffffffff80028e03
#15 [ffff81042deadc78] __make_request at ffffffff8000c22e
#16 [ffff81042deadcf8] generic_make_request at ffffffff8001c465
#17 [ffff81042deadd78] flush_pending_writes at ffffffff88075bc2 [raid1]
#18 [ffff81042deadd98] raid1d at ffffffff88076bbc [raid1]
#19 [ffff81042deade98] md_thread at ffffffff8022062d
#20 [ffff81042deadee8] kthread at ffffffff80032755
#21 [ffff81042deadf48] kernel_thread at ffffffff8005dfb1
crash> kmem -i
PAGES TOTAL PERCENTAGE
TOTAL MEM 12291482 46.9 GB ----
FREE 53347 208.4 MB 0% of TOTAL MEM
USED 12238135 46.7 GB 99% of TOTAL MEM
SHARED 569443 2.2 GB 4% of TOTAL MEM
BUFFERS 45170 176.4 MB 0% of TOTAL MEM
CACHED 109560 428 MB 0% of TOTAL MEM
SLAB 181255 708 MB 1% of TOTAL MEM
TOTAL HIGH 0 0 0% of TOTAL MEM
FREE HIGH 0 0 0% of TOTAL HIGH
TOTAL LOW 12291482 46.9 GB 100% of TOTAL MEM
FREE LOW 53347 208.4 MB 0% of TOTAL LOW
TOTAL SWAP 9437182 36 GB ----
SWAP USED 3591032 13.7 GB 38% of TOTAL SWAP
SWAP FREE 5846150 22.3 GB 61% of TOTAL SWAP
Environment
- Red Hat Enterprise Linux 5.7
- kernel-2.6.18-274.17.1.el5
- Software RAID
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.