System hangs due to xfs lockups

Solution Verified - Updated -

Issue

  • The system goes into a hung state leaving no evidences in the system logs.

  • vmcore shows the following logs of multiple D-state processes:

    [179039.139457] INFO: task varnishd:2941 blocked for more than 120 seconds.
    [179039.139477] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    [179039.139500] varnishd        D 0000000000000000     0  2941   2586 0x00000080
    [179039.139505]  ffff880078177b58 0000000000000086 ffff880078163ec0 ffff880078177fd8
    [179039.139506]  ffff880078177fd8 ffff880078177fd8 ffff880078163ec0 ffff880079f2dc00
    [179039.139507]  ffff880053ffeac8 ffff880079f2ddc0 00000000000009cc 0000000000000000
    [179039.139508] Call Trace:
    [179039.139510]  [<ffffffff8168b879>] schedule+0x29/0x70
    [179039.139522]  [<ffffffffa027d672>] xlog_grant_head_wait+0xb2/0x190 [xfs]
    [179039.139532]  [<ffffffffa027d7ee>] xlog_grant_head_check+0x9e/0x110 [xfs]
    [179039.139542]  [<ffffffffa02812e9>] xfs_log_reserve+0xd9/0x1d0 [xfs]
    [179039.139551]  [<ffffffffa027b84d>] xfs_trans_reserve+0x18d/0x200 [xfs]
    [179039.139562]  [<ffffffffa026c226>] xfs_vn_update_time+0x56/0x190 [xfs]
    [179039.139563]  [<ffffffff81219a75>] update_time+0x25/0xd0
    [179039.139564]  [<ffffffff81200d38>] ? __sb_start_write+0x58/0x110
    [179039.139565]  [<ffffffff81219d20>] file_update_time+0xa0/0xf0
    [179039.139575]  [<ffffffffa026150e>] xfs_filemap_page_mkwrite+0x4e/0x120 [xfs]
    [179039.139577]  [<ffffffff811ac3b4>] do_page_mkwrite+0x54/0xa0
    [179039.139578]  [<ffffffff811aef36>] do_wp_page+0x326/0x530
    [179039.139579]  [<ffffffff811b0f8c>] handle_mm_fault+0x72c/0xfe0
    [179039.139581]  [<ffffffff81691c94>] __do_page_fault+0x154/0x450
    [179039.139583]  [<ffffffff81691fc5>] do_page_fault+0x35/0x90
    [179039.139584]  [<ffffffff8168e288>] page_fault+0x28/0x30
    [187467.575335] audit: audit_backlog=321 > audit_backlog_limit=320
    [187467.575339] audit: audit_lost=1 audit_rate_limit=0 audit_backlog_limit=320
    [187467.575340] audit: backlog limit exceeded
    [187467.575439] audit: audit_backlog=321 > audit_backlog_limit=320
    [187467.575440] audit: audit_lost=2 audit_rate_limit=0 audit_backlog_limit=320
    [187467.575441] audit: backlog limit exceeded
    

Environment

  • Red Hat Enterprise Linux 7

  • XFS Filesystem

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.