GFS2 unmount blocks in dlm_release_lockspace when issuing a reboot of RHEL 6 cluster node

Solution In Progress - Updated -

Issue

  • Occasionally, when one node is shutdown, rgmanager does not exit and we see hung tasks in /var/log/messages for an unmount command waiting in dlm_release_lockspace
  • gfs2 filesystems do not unmount.
  • A hard power shutdown of one node will leave the other node unable to access gfs2 filesystems.
kernel: INFO: task umount:59023 blocked for more than 120 seconds.
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: umount        D 0000000000000006     0 59023  57463 0x00000000
kernel: ffff882026f6dce8 0000000000000082 0000000000000000 0000000000000000
kernel: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
kernel: ffff881291ead058 ffff882026f6dfd8 000000000000fb88 ffff881291ead058
kernel: Call Trace:
kernel: [<ffffffff8150ed3e>] __mutex_lock_slowpath+0x13e/0x180
kernel: [<ffffffff8150ebdb>] mutex_lock+0x2b/0x50
kernel: [<ffffffffa047bea8>] dlm_release_lockspace+0x48/0x480 [dlm]
kernel: [<ffffffffa062b757>] gdlm_unmount+0x27/0x40 [gfs2]
kernel: [<ffffffffa0619821>] gfs2_lm_unmount+0x21/0x30 [gfs2]
kernel: [<ffffffffa0628284>] gfs2_put_super+0x184/0x220 [gfs2]
kernel: [<ffffffff8118326b>] generic_shutdown_super+0x5b/0xe0
kernel: [<ffffffff81183321>] kill_block_super+0x31/0x50
kernel: [<ffffffffa06198a3>] gfs2_kill_sb+0x73/0x80 [gfs2]
kernel: [<ffffffff81183af7>] deactivate_super+0x57/0x80
kernel: [<ffffffff811a1b6f>] mntput_no_expire+0xbf/0x110
kernel: [<ffffffff811a25db>] sys_umount+0x7b/0x3a0
kernel: [<ffffffff81086391>] ? sigprocmask+0x71/0x110
kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b

Environment

  • Red Hat Enterprise Linux (RHEL) 6 Update 4 with the Reslient Storage Add On
    • kernel-2.6.32-358.el6
  • GFS2

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.