raid1d task deadlocked due to memory shortage

Solution Verified - Updated -

Issue

  • System hung and the root cause is believed to be that the raid1d thread is deadlocked.
  • It needs to write pages out to free up memory but can't allocate an I/O request because there's no free memory.
PID: 848    TASK: ffff81042fc8a860  CPU: 5   COMMAND: "md1_raid1"
 #0 [ffff81042dead710] schedule at ffffffff80062fa0
 #1 [ffff81042dead7e8] schedule_timeout at ffffffff8006389f
 #2 [ffff81042dead838] io_schedule_timeout at ffffffff80063241
 #3 [ffff81042dead868] blk_congestion_wait at ffffffff8003b359
 #4 [ffff81042dead8b8] throttle_vm_writeout at ffffffff800535a9
 #5 [ffff81042dead8d8] shrink_zone at ffffffff800132eb
 #6 [ffff81042dead918] try_to_free_pages at ffffffff800ce8ba
 #7 [ffff81042dead9a8] __alloc_pages at ffffffff8000f5f6
 #8 [ffff81042deada18] cache_grow at ffffffff80017aa8
 #9 [ffff81042deada68] cache_alloc_refill at ffffffff8005be11
#10 [ffff81042deadaa8] kmem_cache_alloc at ffffffff8000ad2e
#11 [ffff81042deadac8] mempool_alloc at ffffffff80023469
#12 [ffff81042deadb28] cfq_set_request at ffffffff8001caeb
#13 [ffff81042deadba8] get_request at ffffffff8001361c
#14 [ffff81042deadbf8] get_request_wait at ffffffff80028e03
#15 [ffff81042deadc78] __make_request at ffffffff8000c22e
#16 [ffff81042deadcf8] generic_make_request at ffffffff8001c465
#17 [ffff81042deadd78] flush_pending_writes at ffffffff88075bc2 [raid1]
#18 [ffff81042deadd98] raid1d at ffffffff88076bbc [raid1]
#19 [ffff81042deade98] md_thread at ffffffff8022062d
#20 [ffff81042deadee8] kthread at ffffffff80032755
#21 [ffff81042deadf48] kernel_thread at ffffffff8005dfb1
crash> kmem -i
              PAGES        TOTAL      PERCENTAGE
 TOTAL MEM  12291482      46.9 GB         ----
      FREE    53347     208.4 MB    0% of TOTAL MEM
      USED  12238135      46.7 GB   99% of TOTAL MEM
    SHARED   569443       2.2 GB    4% of TOTAL MEM
   BUFFERS    45170     176.4 MB    0% of TOTAL MEM
    CACHED   109560       428 MB    0% of TOTAL MEM
      SLAB   181255       708 MB    1% of TOTAL MEM

TOTAL HIGH        0            0    0% of TOTAL MEM
 FREE HIGH        0            0    0% of TOTAL HIGH
 TOTAL LOW  12291482      46.9 GB  100% of TOTAL MEM
  FREE LOW    53347     208.4 MB    0% of TOTAL LOW

TOTAL SWAP  9437182        36 GB         ----
 SWAP USED  3591032      13.7 GB   38% of TOTAL SWAP
 SWAP FREE  5846150      22.3 GB   61% of TOTAL SWAP

Environment

  • Red Hat Enterprise Linux 5.7
  • kernel-2.6.18-274.17.1.el5
  • Software RAID

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content