Clustered LVM mirror creation or I/O hangs in RHEL with devices larger than 1.5Tb
Issue
- Creation of new mirrored logical volume in the Red Hat cluster times out:
Error locking on node <nodename>: Command timed out
Failed to activate new LV.
Unable to deactivate failed new LV. Manual intervention required.
- The mirror sync process or I/O to clustered mirror hangs when the device is larger than 1.5Tb
device-mapper: dm-log-clustered: [U9LjlBOE] Request timed out: [DM_CLOG_CTR/22] - retrying
-
pvmoveof clustered volumes greater than 1.5Tb hangs or causes I/O to that device to hang -
While migrating LUNs from one array to another all of a sudden the cluster went in a state of confusion, and I continuously keep getting the
dm-log-clusteredretry and "request timed out" messages, mixed withclogd"sync checkpoint retry messages
Aug 8 09:38:36 node2 device-mapper: dm-log-clustered: [knvU0IKd] Request timed out: [DM_CLOG_RESUME/XXX] - retrying
Aug 8 09:38:36 node1 clogd[10366]: Sync checkpoint section create retry
Aug 8 09:38:36 node1 last message repeated 103 times
Aug 8 09:38:36 node1 kernel: device-mapper: dm-log-clustered: [knvU0IKd] Request timed out: [DM_CLOG_CLEAR_REGION/581] - retrying
Aug 8 09:38:36 node1 clogd[10366]: Sync checkpoint section create retry
Aug 8 09:38:51 node1 last message repeated 7395 times
Environment
- Red Hat Enterprise Linux (RHEL) 5 or 6 with the Resilient Storage Add On
lvm2-cluster- RHEL 6:
lvm2-clusterreleases prior to2.02.111-2.el6_6.2 - RHEL 5: All
lvm2-clusterreleases - One or more volume groups with the clustered attribute set
locking_type = 3in/etc/lvm/lvm.conf
- RHEL 6:
- A cluster mirror segment greater than 1.5Tb or a
pvmoveof a clustered PV greater than 1.5Tb
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
