Mismatch between etcd member IPs and IPs configured for etcd in RHOCP 4

Issue

• Our etcd Pods on our Master Nodes are showing errors like the following:

  2021-01-08T09:14:07.556275311+00:00 stderr F W0108 09:14:07.556103      17 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://10.140.108.21:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.140.108.21:2379: connect: no route to host". Reconnecting...
  [..]
  2021-01-08T09:17:29.499949344+00:00 stderr F 2021-01-08 09:17:29.499921 I | embed: rejected connection from "10.140.108.25:39732" (error "tls: \"10.140.108.25\" does not match any of DNSNames [\"localhost\" \"test45.ocp.example.com\" \"10.140.108.22\"] (lookup test45.ocp.example.com on 10.140.108.23:53: no such host)", ServerName "", IPAddresses ["10.140.108.22"], DNSNames ["localhost" "test45.ocp.example.com" "10.140.108.22"])
  

• etcdctl shows different IP addresses than what is shown in the environment variables of the etcd Pods:

  sh-4.4# etcdctl member list
  26f85dbd1395423d, started, test45-n9lbn-master-1, https://10.140.108.23:2380, https://10.140.108.23:2379, false
  4b8987a866eac5cb, started, test45-n9lbn-master-2, https://10.140.108.21:2380, https://10.140.108.21:2379, false
  c3f7b96cdd81341b, started, test45-n9lbn-master-0, https://10.140.108.22:2380, https://10.140.108.22:2379, false
  sh-4.4# echo $ALL_ETCD_ENDPOINTS
  https://10.140.108.25:2379,https://10.140.108.23:2379,https://10.140.108.21:2379
  

• etcd-ensure-env-vars exits with the following message:

  [root@test45-n9lbn-master-0 ~]# crictl logs 066497679b291
  Expected node IP to be 10.140.108.22 got 10.140.108.25