Upgrading etcd cluster fails - "Verify cluster is healthy"
Issue
Upgrading etcd (3.9 to 3.10 in this case) fails with unhealthy cluster error:
FAILED - RETRYING: Verify cluster is healthy (2 retries left).
FAILED - RETRYING: Verify cluster is healthy (1 retries left).
fatal: [hostname.example.com]: FAILED! => {"attempts": 30, "changed": true, "cmd": ["/usr/local/bin/master-exec", "etcd", "etcd", "etcdctl", "--cert-file", "/etc/etcd/peer.crt", "--key-file", "/etc/etcd/peer.key", "--ca-file", "/etc/etcd/ca.crt", "--endpoints", "https://hostname.example.com:2379", "cluster-health"], "delta": "0:00:00.028552", "end": "2019-05-22 12:58:56.988640", "rc": 0, "start": "2019-05-22 12:58:56.960088", "stderr": "Component etcd is stopped or not running", "stderr_lines": ["Component etcd is stopped or not running"], "stdout": "", "stdout_lines": []}
[WARNING]: Could not create retry file '/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_10/upgrade.retry'. [Errno 21] Is a directory:
u'/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_10/upgrade.retry'
Environment
Red Hat OpenShift Container Platform 3.9
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.