CMAN: too many transition restarts - will die
Issue
- Adding 5th node to running cluster caused full cluster down. Rebooted all nodes fixed the issue.
- Rebooting or fencing a single node in the cluster caused the entire cluster to go down
-
Messages such as the following are seen in
/var/log/messages
Dec 15 16:17:47 hostname kernel: CMAN: too many transition restarts - will die Dec 15 16:17:47 hostname kernel: CMAN: we are leaving the cluster. Inconsistent cluster view Dec 15 16:17:47 hostname kernel: WARNING: dlm_emergency_shutdown Dec 15 16:17:47 hostname kernel: WARNING: dlm_emergency_shutdown Dec 15 16:17:47 hostname kernel: SM: 00000002 sm_stop: SG still joined Dec 15 16:17:47 hostname kernel: SM: 01000003 sm_stop: SG still joined Dec 15 16:17:47 hostname clurgmgrd[10000]: <warning> #67: Shutting down uncleanly Dec 15 16:17:47 hostname kernel: SM: 02000009 sm_stop: SG still joined Dec 15 16:17:47 hostname kernel: SM: 0300001e sm_stop: SG still joined
Environment
- Red Hat Enterprise Linux (RHEL) 4 Update 7 or earlier
- Red Hat Cluster Suite
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.