rgmanager blocking on start and /var/log/messages shows hung task warnings for with dlm_clear_proc_locks in the backtrace in a RHEL 6 High Availability cluster
Issue
- A node was rebooted and it comes up with a hung rgmanager process. It looks like it is blocked
- When starting
rgmanager, it becomes blocked in adevice_closecall in the kernel:
Aug 18 11:30:01 node3 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 18 11:30:01 node3 kernel: rgmanager D 0000000000000001 0 2645 2643 0x00000080
Aug 18 11:30:01 node3 kernel: ffff880138b03cc0 0000000000000082 0000000000000000 ffff880138b03ce8
Aug 18 11:30:01 node3 kernel: ffff880138b03db8 ffff880138b03c48 ffff880138b03c88 ffff8801397b0cc0
Aug 18 11:30:01 node3 kernel: ffff880138b3e638 ffff880138b03fd8 000000000000fbc8 ffff880138b3e638
Aug 18 11:30:01 node3 kernel: Call Trace:
Aug 18 11:30:01 node3 kernel: [<ffffffff8152a295>] rwsem_down_failed_common+0x95/0x1d0
Aug 18 11:30:01 node3 kernel: [<ffffffff8122f854>] ? inode_has_perm+0x54/0xa0
Aug 18 11:30:01 node3 kernel: [<ffffffff8152a426>] rwsem_down_read_failed+0x26/0x30
Aug 18 11:30:01 node3 kernel: [<ffffffff8128e844>] call_rwsem_down_read_failed+0x14/0x30
Aug 18 11:30:01 node3 kernel: [<ffffffff81529924>] ? down_read+0x24/0x30
Aug 18 11:30:01 node3 kernel: [<ffffffffa02a148d>] dlm_clear_proc_locks+0x3d/0x2a0 [dlm]
Aug 18 11:30:01 node3 kernel: [<ffffffffa02acb36>] device_close+0x66/0xc0 [dlm]
Aug 18 11:30:01 node3 kernel: [<ffffffff8118a715>] __fput+0xf5/0x210
Aug 18 11:30:01 node3 kernel: [<ffffffff8118a855>] fput+0x25/0x30
Aug 18 11:30:01 node3 kernel: [<ffffffff81185b7d>] filp_close+0x5d/0x90
Aug 18 11:30:01 node3 kernel: [<ffffffff81185c55>] sys_close+0xa5/0x100
Aug 18 11:30:01 node3 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Environment
- Red Hat Enterprise Linux (RHEL) 6 with the High Availability Add On
rgmanager
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Close
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.