clvmd seems to be using more resources than expected or clustered lvm commands are running slowly when using Pacemaker in RHEL 6
Environment
- Red Hat Enterprise Linux (RHEL) 6 with the Resilient Storage Add On
Pacemaker
lvm2-cluster
locking_type = 3
in/etc/lvm/lvm.conf
- A
lsb:clvmd
resource configured inPacemaker
to start/stop/monitor theclvmd
daemon
Issue
clvmd
seems to be using more CPU than expected- LVM commands are slow when using
Pacemaker
and aclvmd
resource
Resolution
Red Hat does not recommend or support managing clvmd
as a resource via pacemaker
in RHEL 6. It is recommended that clvmd
be started directly using the init service (service clvmd start/stop/status
, or chkconfig clvmd on
).
If clvmd
is being managed by pacemaker
despite these recommendations, set the monitor interval to 60s or higher to avoid these performance issues.
# pcs resource create clvmd lsb:clvmd op monitor interval=60s on-fail=fence --clone
Or if the resource is already created (assuming the resource-id is "clvmd"):
# pcs resource update clvmd op monitor interval=60s
Root Cause
Pacemaker
offers the ability to configure init services like clvmd
as cluster resources so they can be started and monitored according to defined policies. However, Red Hat does not recommend doing so for cluster-related daemons in RHEL 6.
When creating such a resource, the interval on the monitor operation can be set. If this interval is set too low, it will cause frequent status checks on the clvmd
daemon which can trigger excessive activity across the cluster as it processes the state of volumes, syncs locks, etc. If for some reason the clvmd
service must be managed by a resource, it is recommended to set this to at least 60 seconds or higher to avoid issues such as this.
Diagnostic Steps
- Use
pcs
to determine the monitor interval for theclvmd
resource (assuming the resource-id for the resource is "clvmd"):
# pcs resource show clvmd
Resource: clvmd (class=lsb type=clvmd)
Operations: monitor interval=5s (clvmd-monitor-interval-5s)
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments