clvmd seems to be using more resources than expected or clustered lvm commands are running slowly when using Pacemaker in RHEL 6

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 6 with the Resilient Storage Add On
  • Pacemaker
  • lvm2-cluster
    • locking_type = 3 in /etc/lvm/lvm.conf
  • A lsb:clvmd resource configured in Pacemaker to start/stop/monitor the clvmd daemon

Issue

  • clvmd seems to be using more CPU than expected
  • LVM commands are slow when using Pacemaker and a clvmd resource

Resolution

Red Hat does not recommend or support managing clvmd as a resource via pacemaker in RHEL 6. It is recommended that clvmd be started directly using the init service (service clvmd start/stop/status, or chkconfig clvmd on).

If clvmd is being managed by pacemaker despite these recommendations, set the monitor interval to 60s or higher to avoid these performance issues.

# pcs resource create clvmd lsb:clvmd op monitor interval=60s on-fail=fence --clone

Or if the resource is already created (assuming the resource-id is "clvmd"):

# pcs resource update clvmd op monitor interval=60s

Root Cause

Pacemaker offers the ability to configure init services like clvmd as cluster resources so they can be started and monitored according to defined policies. However, Red Hat does not recommend doing so for cluster-related daemons in RHEL 6.

When creating such a resource, the interval on the monitor operation can be set. If this interval is set too low, it will cause frequent status checks on the clvmd daemon which can trigger excessive activity across the cluster as it processes the state of volumes, syncs locks, etc. If for some reason the clvmd service must be managed by a resource, it is recommended to set this to at least 60 seconds or higher to avoid issues such as this.

Diagnostic Steps

  • Use pcs to determine the monitor interval for the clvmd resource (assuming the resource-id for the resource is "clvmd"):
# pcs resource show clvmd
 Resource: clvmd (class=lsb type=clvmd)
  Operations: monitor interval=5s (clvmd-monitor-interval-5s)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.