rgmanager blocking on start and /var/log/messages shows hung task warnings for with dlm_clear_proc_locks in the backtrace in a RHEL 6 High Availability cluster

Issue

• A node was rebooted and it comes up with a hung rgmanager process. It looks like it is blocked
• When starting rgmanager, it becomes blocked in a device_close call in the kernel:

Aug 18 11:30:01 node3 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 18 11:30:01 node3 kernel: rgmanager     D 0000000000000001     0  2645   2643 0x00000080
Aug 18 11:30:01 node3 kernel: ffff880138b03cc0 0000000000000082 0000000000000000 ffff880138b03ce8
Aug 18 11:30:01 node3 kernel: ffff880138b03db8 ffff880138b03c48 ffff880138b03c88 ffff8801397b0cc0
Aug 18 11:30:01 node3 kernel: ffff880138b3e638 ffff880138b03fd8 000000000000fbc8 ffff880138b3e638
Aug 18 11:30:01 node3 kernel: Call Trace:
Aug 18 11:30:01 node3 kernel: [<ffffffff8152a295>] rwsem_down_failed_common+0x95/0x1d0
Aug 18 11:30:01 node3 kernel: [<ffffffff8122f854>] ? inode_has_perm+0x54/0xa0
Aug 18 11:30:01 node3 kernel: [<ffffffff8152a426>] rwsem_down_read_failed+0x26/0x30
Aug 18 11:30:01 node3 kernel: [<ffffffff8128e844>] call_rwsem_down_read_failed+0x14/0x30
Aug 18 11:30:01 node3 kernel: [<ffffffff81529924>] ? down_read+0x24/0x30
Aug 18 11:30:01 node3 kernel: [<ffffffffa02a148d>] dlm_clear_proc_locks+0x3d/0x2a0 [dlm]
Aug 18 11:30:01 node3 kernel: [<ffffffffa02acb36>] device_close+0x66/0xc0 [dlm]
Aug 18 11:30:01 node3 kernel: [<ffffffff8118a715>] __fput+0xf5/0x210
Aug 18 11:30:01 node3 kernel: [<ffffffff8118a855>] fput+0x25/0x30
Aug 18 11:30:01 node3 kernel: [<ffffffff81185b7d>] filp_close+0x5d/0x90
Aug 18 11:30:01 node3 kernel: [<ffffffff81185c55>] sys_close+0xa5/0x100
Aug 18 11:30:01 node3 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b