I/O blocked to clustered mirror after "dm-log-clustered: Userspace cluster log server not found" and "clogd[XXXXX]: Failed to recv message from kernel" in RHEL 5
Issue
- I/O to a cluster mirrored logical volume blocks after we see repeated time outs, followed by "Userspace cluster log server not found." and "Failed to recv message from kernel"
Nov 29 05:40:18 node1 kernel: device-mapper: dm-log-clustered: [VmUKhcE4] Request timed out: [DM_CLOG_GET_SYNC_COUNT/42516260] - retrying
Nov 29 05:40:18 node1 kernel: device-mapper: dm-log-clustered: [aVgPDEwC] Request timed out: [DM_CLOG_STATUS_TABLE/42516261] - retrying
Nov 29 05:40:18 node1 kernel: device-mapper: dm-log-clustered: Unable to send cluster log request [DM_CLOG_STATUS_TABLE] to server: -3
Nov 29 05:40:18 node1 kernel: device-mapper: dm-log-clustered: Userspace cluster log server not found.
Nov 29 05:40:18 node1 clogd[31940]: Failed to recv message from kernel
Nov 29 05:40:19 node1 kernel: device-mapper: dm-log-clustered: [VGEcXSZi] Request timed out: [DM_CLOG_IN_SYNC/42516262] - retrying
Nov 29 05:40:19 node1 kernel: device-mapper: dm-log-clustered: Unable to send cluster log request [DM_CLOG_IN_SYNC] to server: -3
Nov 29 05:40:19 node1 kernel: device-mapper: dm-log-clustered: Userspace cluster log server not found.
Nov 29 05:40:20 node1 kernel: device-mapper: dm-log-clustered: [3JEDH8Ef] Request timed out: [DM_CLOG_MARK_REGION/42516263] - retrying
Nov 29 05:40:20 node1 kernel: device-mapper: dm-log-clustered: Unable to send cluster log request [DM_CLOG_MARK_REGION] to server: -3
Nov 29 05:40:20 node1 kernel: device-mapper: dm-log-clustered: Userspace cluster log server not found.
Nov 29 05:40:20 node1 kernel: device-mapper: dm-log-clustered: Attempting to contact cluster log server...
Environment
- Red Hat Enterprise Linux (RHEL) 5 with the Resilient Storage Add On
- Mirrored logical volume(s) in a clustered volume group
cmirrorlvm2-clusterlocking_type = 3in/etc/lvm/lvm.conf- Clustered attribute set on volume group
- One or more mirrored LVs in clustered volume group
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.