RHEL7: XFS deadlocked with xfs_buf inconsistent flags (XBF_ASYNC on a synchronous I/O request)

Issue

• system hang with following logs.

[34806338.022488] INFO: task kswapd1:443 blocked for more than 120 seconds.
[34806338.022512] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[34806338.022533] kswapd1         D 0000000000000000     0   443      2 0x00000000
[34806338.022536]  ffff88084ea778f0 0000000000000046 ffff881051fad080 ffff88084ea77fd8
[34806338.022537]  ffff88084ea77fd8 ffff88084ea77fd8 ffff881051fad080 ffff88081bfa40d8
[34806338.022539]  ffff88081bfa40e0 7fffffffffffffff ffff881051fad080 0000000000000000
[34806338.022540] Call Trace:
[34806338.022545]  [<ffffffff8163c229>] schedule+0x29/0x70
[34806338.022547]  [<ffffffff81639f19>] schedule_timeout+0x209/0x2d0
[34806338.022550]  [<ffffffff812cc6e4>] ? blk_finish_plug+0x14/0x40
[34806338.022587]  [<ffffffffa03551c4>] ? _xfs_buf_ioapply+0x334/0x460 [xfs]
[34806338.022588]  [<ffffffff8163c5f6>] wait_for_completion+0x116/0x170
[34806338.022591]  [<ffffffff810b8d30>] ? wake_up_state+0x20/0x20
[34806338.022601]  [<ffffffffa0357124>] ? xfs_bwrite+0x24/0x60 [xfs]
[34806338.022610]  [<ffffffffa0356cd4>] xfs_buf_submit_wait+0x84/0x1d0 [xfs]
[34806338.022618]  [<ffffffffa0357124>] xfs_bwrite+0x24/0x60 [xfs]
[34806338.022627]  [<ffffffffa035ecdb>] xfs_reclaim_inode+0x32b/0x350 [xfs]
[34806338.022635]  [<ffffffffa035ef67>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[34806338.022637]  [<ffffffff810b5ea9>] ? ttwu_do_wakeup+0x19/0xd0
[34806338.022639]  [<ffffffff810b603d>] ? ttwu_do_activate.constprop.84+0x5d/0x70
[34806338.022640]  [<ffffffff810b8b86>] ? try_to_wake_up+0x1b6/0x300
[34806338.022642]  [<ffffffff810b8cf3>] ? wake_up_process+0x23/0x40
[34806338.022651]  [<ffffffffa035fb73>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[34806338.022660]  [<ffffffffa036eae5>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[34806338.022663]  [<ffffffff811e1628>] prune_super+0xe8/0x170
[34806338.022665]  [<ffffffff8117cb65>] shrink_slab+0x165/0x300
[34806338.022667]  [<ffffffff811d6897>] ? vmpressure+0x87/0x90
[34806338.022668]  [<ffffffff811807d1>] balance_pgdat+0x4b1/0x5e0
[34806338.022669]  [<ffffffff81180a73>] kswapd+0x173/0x450
[34806338.022671]  [<ffffffff810a6b80>] ? wake_up_atomic_t+0x30/0x30
[34806338.022672]  [<ffffffff81180900>] ? balance_pgdat+0x5e0/0x5e0
[34806338.022674]  [<ffffffff810a5b8f>] kthread+0xcf/0xe0
[34806338.022676]  [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
[34806338.022678]  [<ffffffff81647158>] ret_from_fork+0x58/0x90
[34806338.022680]  [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140