Low memory deadlock with md devices and external (imsm) metadata handling

Solution Unverified - Updated -

Issue

  • The server got hung up. Ran sysrq-c to capture vmcore.

  • Lots of tasks are waiting on XFS I/O for page reclaiming with the backtrace like this:


PID: 36781  TASK: ffff96bff2f330c0  CPU: 13  COMMAND: "splunkd"
 #0 [ffff96c6dbab7578] __schedule at ffffffff8a567747
 #1 [ffff96c6dbab7608] schedule at ffffffff8a567c49
 #2 [ffff96c6dbab7618] _xfs_log_force_lsn at ffffffffc0839fe4 [xfs]
 #3 [ffff96c6dbab76a0] xfs_log_force_lsn at ffffffffc083a1a4 [xfs]
 #4 [ffff96c6dbab76d8] __xfs_iunpin_wait at ffffffffc0827c2c [xfs]
 #5 [ffff96c6dbab7750] xfs_iunpin_wait at ffffffffc082af89 [xfs]
 #6 [ffff96c6dbab7760] xfs_reclaim_inode at ffffffffc081e7d3 [xfs]
 #7 [ffff96c6dbab77b0] xfs_reclaim_inodes_ag at ffffffffc081ec57 [xfs]
 #8 [ffff96c6dbab7948] xfs_reclaim_inodes_nr at ffffffffc081fe33 [xfs]
 #9 [ffff96c6dbab7968] xfs_fs_free_cached_objects at ffffffffc0830445 [xfs]
#10 [ffff96c6dbab7978] prune_super at ffffffff8a044286
#11 [ffff96c6dbab79b0] shrink_slab at ffffffff89fc9de5
#12 [ffff96c6dbab7a50] do_try_to_free_pages at ffffffff89fcd0a2
#13 [ffff96c6dbab7ac8] try_to_free_pages at ffffffff89fcd2bc
#14 [ffff96c6dbab7b60] __alloc_pages_slowpath at ffffffff8a55d1ef
#15 [ffff96c6dbab7c50] __alloc_pages_nodemask at ffffffff89fc1145
#16 [ffff96c6dbab7d00] alloc_pages_vma at ffffffff8a011535
#17 [ffff96c6dbab7d70] handle_pte_fault at ffffffff89fe9a77
#18 [ffff96c6dbab7e08] handle_mm_fault at ffffffff89fec01d
#19 [ffff96c6dbab7eb0] __do_page_fault at ffffffff8a56f5e3
#20 [ffff96c6dbab7f20] do_page_fault at ffffffff8a56f915
#21 [ffff96c6dbab7f50] page_fault at ffffffff8a56b758
    RIP: 0000557a7a3c7318  RSP: 00007f4714dfeb60  RFLAGS: 00010297
    RAX: 00007f4c849a6000  RBX: 00007f4ff4616000  RCX: 0000000000116b71
    RDX: 00007f4c84eb5b88  RSI: 00007f458ae00000  RDI: 000000000007ffff
    RBP: 0000000000116b71   R8: 00007f50011305a0   R9: 0000000000000268
    R10: 00007f4ff460e000  R11: 00007f4ff460e000  R12: 00000000001975ed
    R13: 0000000000008098  R14: 0000000000040000  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0033  SS: 002b

Environment

  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 6
  • Software RAID (mdraid)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content