performance or connectivity issues in RHEL when using cmirror

Solution Verified - Updated -

Issue

  • a node is evicted or fenced after several minutes of retry messages from dm-log-clustered such as:
  Mar  2 07:25:26 node2 kernel: device-mapper: dm-log-clustered: [aETjx3eT] Request timed out: [DM_CLOG_MARK_REGION/203943964] - retrying
  Mar  2 07:25:26 node1 kernel: device-mapper: dm-log-clustered: [La2wV73X] Request timed out: [DM_CLOG_MARK_REGION/203943968] - retrying
  • cmirror severely impacts performance
  • LVM mirroring seems to take an inordinate amount of IO capacity, slowing the machines and having a severe impact on the applications running
  • As soon as clvmd is started, all three nodes begin to resync the mirrors in the volume groups, and during this resync, the IO load is such that restarting the actual applications with a useful performance is difficult, and rgmanager cannot be started to let the applications recover until the resync is completed.
  • Performance degradation occurs when converting a logical volume to mirrored, or writing large amounts of data to a cluster mirror.

Environment

  • Red Hat Enterprise Linux (RHEL) 5, 6, or 7 with the Resilient Storage Add On
  • lvm2-cluster
    • locking_type = 3 in /etc/lvm/lvm.conf
    • One or more volume groups with the 'c' (clustered) attribute set
  • cmirror
    • One or more clustered volume groups with a mirrored logical volume in it

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content