The temporary file /tmp/cmanstartup.XXXX will continue to grow in size after it was removed on Red Hat Enterprise Linux 6
Environment
- Red Hat Enterprise Linux Server 6 (with the High Availability Add on)
- cluster-3.0.12-41.el6_1.2 and earlier packages.
- Cluster debug logging is enabled in /etc/cluster/cluster.conf:
$ grep logging /etc/cluster/cluster.conf
<logging debug="on"/>
Issue
- The file /tmp/cmanstartup.XXXX is not removed completely(only unlinked) when cman is started. The file continues to grows and will grow faster if debugging is enabled in the /etc/cluster/cluster.conf file:
$ grep logging /etc/cluster/cluster.conf
<logging debug="on"/>
Resolution
- Update to cluster-3.0.12.1-23.el6 (from RHBA-2011:1516) or later, which contains a fix for this issue.
- Corosync must be restarted to release the deleted file. The easiest way to do this is to schedule a rolling-restart of cman on all cluster nodes, or simply reboot each node.
Root Cause
- The issue is that the file /tmp/cmanstartup.XXXX does not exist because it was removed by the /etc/init.d/cman script after a start or stop operation. If cman is started successfully then the corosync process continues to write to that file even though it was unlinked (which is normal). The file which is now a zombie file continues to grow even thought it does not exist on the filesystem.
Diagnostic Steps
-
If the Environment listed above matches following symptoms are present on your system, then this article may match your issue:
-
List all the open files and grep for the deleted ones while cman is started. The /tmp/cmanstartup.XXXX file should be listed and the file should continue to grow:
$ lsof /tmp | grep -i deleted
corosync 3549 root 2w REG 253,5 294628530 13 /tmp/cmanstartup.tUWDjQpjhf (deleted)
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
