Red Hat Training

A Red Hat training course is available for OpenShift Container Platform

Chapter 39. Adding etcd nodes after restoring

After the first instance is running, you can add multiple etcd servers to your cluster.

Procedure

  1. Get the etcd name for the instance in the ETCD_NAME variable: 

    # grep ETCD_NAME /etc/etcd/etcd.conf

  2. Get the IP address where etcd listens for peer communication: 

    # grep ETCD_INITIAL_ADVERTISE_PEER_URLS /etc/etcd/etcd.conf

  3. If the node was previously part of a etcd cluster, delete the previous etcd data: 

    # rm -Rf /var/lib/etcd/*

  4. On the etcd host where etcd is properly running, add the new member:

    • If you use the v2 etcd api, run this command: 

      $ etcdctl2 member add <name> <advertise_peer_urls>

    • If you use the v3 etcd api, run this command: 

      # etcdctl3 member add *<name>* \
  --peer-urls="*<advertise_peer_urls>*"

    The command outputs some variables. For example: 

    ETCD_NAME="master2"
ETCD_INITIAL_CLUSTER="master-0.example.com=https://192.168.55.8:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

  5. Add the values from the previous command to the /etc/etcd/etcd.conf file of the new host: 

    # vi /etc/etc/etcd.conf

  6. Start the etcd service in the node joining the cluster: 

    # systemctl start etcd.service

  7. Check for error messages: 

    $ journalctl -fu etcd.service
  8. Repeat the previous steps for every etcd node to be added.

  9. Once you add all the nodes, verify the cluster status and cluster health:

    • If you use the v2 etcd api, run the following commands: 

      # etcdctl2 member list
5cd050b4d701: name=master1 peerURLs=https://10.0.0.7:2380 clientURLs=https://10.0.0.7:2379 isLeader=true
d0c57659d8990cbd: name=master2 peerURLs=https://10.0.0.5:2380 clientURLs=https://10.0.0.5:2379 isLeader=false
e4696d637de3eb2d: name=master3 peerURLs=https://10.0.0.6:2380 clientURLs=https://10.0.0.6:2379 isLeader=false
      # etcdctl2 cluster-health
member 5cd050b4d701 is healthy: got healthy result from https://10.0.0.7:2379
member d0c57659d8990cbd is healthy: got healthy result from https://10.0.0.5:2379
member e4696d637de3eb2d is healthy: got healthy result from https://10.0.0.6:2379
cluster is healthy

    • If you use the v3 etcd api, run the following commands: 

      # etcdctl3 endpoint health
https://master-0.example.com:2379 is healthy: successfully committed proposal: took = 1.423459ms
https://master-1.example.com:2379 is healthy: successfully committed proposal: took = 1.767481ms
https://master-2.example.com:2379 is healthy: successfully committed proposal: took = 1.599694ms

# etcdctl3 endpoint status
https://master-0.example.com:2379, 40bef1f6c79b3163, 3.2.5, 28 MB, true, 9, 2878
https://master-1.example.com:2379, 1ea57201a3ff620a, 3.2.5, 28 MB, false, 9, 2878
https://master-2.example.com:2379, 59229711e4bc65c8, 3.2.5, 28 MB, false, 9, 2878