clusvcadm command hangs, clustat does not show service states, or other rgmanager-based operations block on a RHEL High Availability cluster node

Issue

• clusvcadm -d, clusvcadm -e, or other clusvcadm operations hang and never return
• When running the clustat command we do not see any services, and it lists the following error message:

Service states unavailable: Temporary failure; try again

• clurgmgrd resgroup_thread_main stuck waiting on dlm lock for service, causing other operations (such as disabling services) to hang
• I can't enable or disable services. The operation just hangs forever.
• rgmanager is hung.
• clustat or rgmanager hangs and doesn't provide service lists
• Nodes can talk to each other, both can see online but while we fire clustat, services are not listing.
• clustat reports "Timed out waiting for a response from Resource Group Manager"
• When I start rgmanager, there are no errors but it never seems to finish starting and doesn't log anything useful, and doesn't start services or initialize resources.
• You see this in clustat:

Timed out waiting for a response from Resource Group Manager

• There are four services running in this cluster, and we're trying to freeze each one with clusvcadm -Z. All services except one get frozen, the other one just sits there forever without returning.
• clusvcamd is hanging and not showing any detail while disabling the service