clusvcadm command hangs, clustat does not show service states, or other rgmanager-based operations block on a RHEL High Availability cluster node

Solution Verified - Updated -

Issue

  • clusvcadm -d, clusvcadm -e, or other clusvcadm operations hang and never return
  • When running the clustat command we do not see any services, and it lists the following error message:
Service states unavailable: Temporary failure; try again
  • clurgmgrd resgroup_thread_main stuck waiting on dlm lock for service, causing other operations (such as disabling services) to hang
  • I can't enable or disable services. The operation just hangs forever.
  • rgmanager is hung.
  • clustat or rgmanager hangs and doesn't provide service lists
  • Nodes can talk to each other, both can see online but while we fire clustat, services are not listing.
  • clustat reports "Timed out waiting for a response from Resource Group Manager"
  • When I start rgmanager, there are no errors but it never seems to finish starting and doesn't log anything useful, and doesn't start services or initialize resources.
  • You see this in clustat:
Timed out waiting for a response from Resource Group Manager
  • There are four services running in this cluster, and we're trying to freeze each one with clusvcadm -Z. All services except one get frozen, the other one just sits there forever without returning.
  • clusvcamd is hanging and not showing any detail while disabling the service

Environment

  • Red Hat Enterprise Linux (RHEL) 5 or 6 with the High Availability Add On
  • rgmanager

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content