raid1d task deadlocked due to memory shortage
Issue
- System hung and the root cause is believed to be that the raid1d thread is deadlocked.
- It needs to write pages out to free up memory but can't allocate an I/O request because there's no free memory.
PID: 848 TASK: ffff81042fc8a860 CPU: 5 COMMAND: "md1_raid1"
#0 [ffff81042dead710] schedule at ffffffff80062fa0
#1 [ffff81042dead7e8] schedule_timeout at ffffffff8006389f
#2 [ffff81042dead838] io_schedule_timeout at ffffffff80063241
#3 [ffff81042dead868] blk_congestion_wait at ffffffff8003b359
#4 [ffff81042dead8b8] throttle_vm_writeout at ffffffff800535a9
#5 [ffff81042dead8d8] shrink_zone at ffffffff800132eb
#6 [ffff81042dead918] try_to_free_pages at ffffffff800ce8ba
#7 [ffff81042dead9a8] __alloc_pages at ffffffff8000f5f6
#8 [ffff81042deada18] cache_grow at ffffffff80017aa8
#9 [ffff81042deada68] cache_alloc_refill at ffffffff8005be11
#10 [ffff81042deadaa8] kmem_cache_alloc at ffffffff8000ad2e
#11 [ffff81042deadac8] mempool_alloc at ffffffff80023469
#12 [ffff81042deadb28] cfq_set_request at ffffffff8001caeb
#13 [ffff81042deadba8] get_request at ffffffff8001361c
#14 [ffff81042deadbf8] get_request_wait at ffffffff80028e03
#15 [ffff81042deadc78] __make_request at ffffffff8000c22e
#16 [ffff81042deadcf8] generic_make_request at ffffffff8001c465
#17 [ffff81042deadd78] flush_pending_writes at ffffffff88075bc2 [raid1]
#18 [ffff81042deadd98] raid1d at ffffffff88076bbc [raid1]
#19 [ffff81042deade98] md_thread at ffffffff8022062d
#20 [ffff81042deadee8] kthread at ffffffff80032755
#21 [ffff81042deadf48] kernel_thread at ffffffff8005dfb1
crash> kmem -i
PAGES TOTAL PERCENTAGE
TOTAL MEM 12291482 46.9 GB ----
FREE 53347 208.4 MB 0% of TOTAL MEM
USED 12238135 46.7 GB 99% of TOTAL MEM
SHARED 569443 2.2 GB 4% of TOTAL MEM
BUFFERS 45170 176.4 MB 0% of TOTAL MEM
CACHED 109560 428 MB 0% of TOTAL MEM
SLAB 181255 708 MB 1% of TOTAL MEM
TOTAL HIGH 0 0 0% of TOTAL MEM
FREE HIGH 0 0 0% of TOTAL HIGH
TOTAL LOW 12291482 46.9 GB 100% of TOTAL MEM
FREE LOW 53347 208.4 MB 0% of TOTAL LOW
TOTAL SWAP 9437182 36 GB ----
SWAP USED 3591032 13.7 GB 38% of TOTAL SWAP
SWAP FREE 5846150 22.3 GB 61% of TOTAL SWAP
Environment
- Red Hat Enterprise Linux 5.7
- kernel-2.6.18-274.17.1.el5
- Software RAID
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
