I/O blocked to clustered mirror after "dm-log-clustered: Userspace cluster log server not found" and "clogd[XXXXX]: Failed to recv message from kernel" in RHEL 5

Solution In Progress - Updated -

Issue

  • I/O to a cluster mirrored logical volume blocks after we see repeated time outs, followed by "Userspace cluster log server not found." and "Failed to recv message from kernel"
Nov 29 05:40:18 node1 kernel: device-mapper: dm-log-clustered: [VmUKhcE4] Request timed out: [DM_CLOG_GET_SYNC_COUNT/42516260] - retrying
Nov 29 05:40:18 node1 kernel: device-mapper: dm-log-clustered: [aVgPDEwC] Request timed out: [DM_CLOG_STATUS_TABLE/42516261] - retrying
Nov 29 05:40:18 node1 kernel: device-mapper: dm-log-clustered: Unable to send cluster log request [DM_CLOG_STATUS_TABLE] to server: -3
Nov 29 05:40:18 node1 kernel: device-mapper: dm-log-clustered:  Userspace cluster log server not found.                
Nov 29 05:40:18 node1 clogd[31940]: Failed to recv message from kernel                                                 
Nov 29 05:40:19 node1 kernel: device-mapper: dm-log-clustered: [VGEcXSZi] Request timed out: [DM_CLOG_IN_SYNC/42516262] - retrying
Nov 29 05:40:19 node1 kernel: device-mapper: dm-log-clustered: Unable to send cluster log request [DM_CLOG_IN_SYNC] to server: -3
Nov 29 05:40:19 node1 kernel: device-mapper: dm-log-clustered:  Userspace cluster log server not found.
Nov 29 05:40:20 node1 kernel: device-mapper: dm-log-clustered: [3JEDH8Ef] Request timed out: [DM_CLOG_MARK_REGION/42516263] - retrying
Nov 29 05:40:20 node1 kernel: device-mapper: dm-log-clustered: Unable to send cluster log request [DM_CLOG_MARK_REGION] to server: -3
Nov 29 05:40:20 node1 kernel: device-mapper: dm-log-clustered:  Userspace cluster log server not found.
Nov 29 05:40:20 node1 kernel: device-mapper: dm-log-clustered: Attempting to contact cluster log server...

Environment

  • Red Hat Enterprise Linux (RHEL) 5 with the Resilient Storage Add On
  • Mirrored logical volume(s) in a clustered volume group
    • cmirror
    • lvm2-cluster
    • locking_type = 3 in /etc/lvm/lvm.conf
    • Clustered attribute set on volume group
    • One or more mirrored LVs in clustered volume group

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.