High Availability cluster shows non-existent estranged nodes in clustat output, or has "Received message has invalid digest... ignoring" messages in logs

Solution Verified - Updated -

Issue

  • A cluster node has many messages like the following in /var/log/messages:

    Jan 24 04:08:25 node1 openais[743]: [MAIN ] Received message has invalid digest... ignoring.  
    Jan 24 04:08:25 node1 openais[743]: [MAIN ] Invalid packet data
    - OR -
    Jan 24 04:08:25 node1 corosync[743]: [MAIN ] Received message has invalid digest... ignoring.  
    Jan 24 04:08:25 node1 corosync[743]: [MAIN ] Invalid packet data
    
  • clustat output shows estranged nodes in the node list that aren't a part of the cluster:

    Cluster Status for test-cluster @ Wed Oct  7 10:18:38 2009
    Member Status: Quorate
    
     Member Name                             ID   Status
     ------ ----                             ---- ------
     node1.test                                  2 Online, Local
     node2.test                                  1 Online
     node7.prod                                  7 Offline, Estranged
    ~~~
    
  • Cluster shows odd behavior such as pulling the wrong configuration file or config_version number when using cman_tool version -r or other synchronization mechanism.

Environment

  • Red Hat Enterprise Linux 5, 6, 7, or 8 (with the High Availability Add-on)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In