RHEL7: XFS deadlocked with xfs_buf inconsistent flags (XBF_ASYNC on a synchronous I/O request)

Solution Verified - Updated -

Issue

  • system hang with following logs.
[34806338.022488] INFO: task kswapd1:443 blocked for more than 120 seconds.
[34806338.022512] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[34806338.022533] kswapd1         D 0000000000000000     0   443      2 0x00000000
[34806338.022536]  ffff88084ea778f0 0000000000000046 ffff881051fad080 ffff88084ea77fd8
[34806338.022537]  ffff88084ea77fd8 ffff88084ea77fd8 ffff881051fad080 ffff88081bfa40d8
[34806338.022539]  ffff88081bfa40e0 7fffffffffffffff ffff881051fad080 0000000000000000
[34806338.022540] Call Trace:
[34806338.022545]  [<ffffffff8163c229>] schedule+0x29/0x70
[34806338.022547]  [<ffffffff81639f19>] schedule_timeout+0x209/0x2d0
[34806338.022550]  [<ffffffff812cc6e4>] ? blk_finish_plug+0x14/0x40
[34806338.022587]  [<ffffffffa03551c4>] ? _xfs_buf_ioapply+0x334/0x460 [xfs]
[34806338.022588]  [<ffffffff8163c5f6>] wait_for_completion+0x116/0x170
[34806338.022591]  [<ffffffff810b8d30>] ? wake_up_state+0x20/0x20
[34806338.022601]  [<ffffffffa0357124>] ? xfs_bwrite+0x24/0x60 [xfs]
[34806338.022610]  [<ffffffffa0356cd4>] xfs_buf_submit_wait+0x84/0x1d0 [xfs]
[34806338.022618]  [<ffffffffa0357124>] xfs_bwrite+0x24/0x60 [xfs]
[34806338.022627]  [<ffffffffa035ecdb>] xfs_reclaim_inode+0x32b/0x350 [xfs]
[34806338.022635]  [<ffffffffa035ef67>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[34806338.022637]  [<ffffffff810b5ea9>] ? ttwu_do_wakeup+0x19/0xd0
[34806338.022639]  [<ffffffff810b603d>] ? ttwu_do_activate.constprop.84+0x5d/0x70
[34806338.022640]  [<ffffffff810b8b86>] ? try_to_wake_up+0x1b6/0x300
[34806338.022642]  [<ffffffff810b8cf3>] ? wake_up_process+0x23/0x40
[34806338.022651]  [<ffffffffa035fb73>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[34806338.022660]  [<ffffffffa036eae5>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[34806338.022663]  [<ffffffff811e1628>] prune_super+0xe8/0x170
[34806338.022665]  [<ffffffff8117cb65>] shrink_slab+0x165/0x300
[34806338.022667]  [<ffffffff811d6897>] ? vmpressure+0x87/0x90
[34806338.022668]  [<ffffffff811807d1>] balance_pgdat+0x4b1/0x5e0
[34806338.022669]  [<ffffffff81180a73>] kswapd+0x173/0x450
[34806338.022671]  [<ffffffff810a6b80>] ? wake_up_atomic_t+0x30/0x30
[34806338.022672]  [<ffffffff81180900>] ? balance_pgdat+0x5e0/0x5e0
[34806338.022674]  [<ffffffff810a5b8f>] kthread+0xcf/0xe0
[34806338.022676]  [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
[34806338.022678]  [<ffffffff81647158>] ret_from_fork+0x58/0x90
[34806338.022680]  [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140

Environment

  • Red Hat Enterprise Linux (RHEL) 7
  • affected kernels
    • rhel7.2: kernels after (including) 3.10.0-327.el7 and before 3.10.0-327.98.2.el7
    • rhel7.3: kernels after (including) 3.10.0-514.el7 and before 3.10.0-514.32.2.el7
    • rhel7.4: kernels after (including) 3.10.0-693.el7 and before 3.10.0-693.2.1.el7

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content