Unable to Access the Cluster via Console and CLI Due to Missing LB Entry in RHOCP 4

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • F5 External LoadBalancer

Issue

  • Cluster access via CLI and web console is unavailable due to connectivity failures to the API and OAuth endpoints.

Resolution

Update the External LoadBalancer configuration with the required entries to restore cluster accessibility.

Root Cause

Certain endpoint entries are not present in the External LoadBalancer.

Diagnostic Steps

  • Verify curl connectivity to the api, api-int, and oauth URLs:
$ curl -vk https://api.<cluster-openshift-domain>:6443
* About to connect() to api.<cluster-openshift-domain> port 6443 (#0)
*   Trying 10.xx.xxx.xx...
* No route to host
* Failed connect to api.<cluster-openshift-domain>:6443; No route to host
* Closing connection 0
curl: (7) Failed connect to api.<cluster-openshift-domain>:6443; No route to host

$ curl -kv https://oauth-openshift.apps.<cluster-openshift-domain>:443
*   Trying 10.15.xxx.xxx:xxx...
* connect to 10.15.xxx.xxx port 443 failed: Connection refused
* Failed to connect to oauth-openshift.apps.<cluster-openshift-domain> port 443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to oauth-openshift.apps.<cluster-openshift-domain> port 443: Connection refused
  • Verify the curl connectivity after bypassing the LB:
$ curl -kvvv --resolve api-int.<cluster-openshift-domain>:6443:<infra_node_ip> https://api-int.<cluster-openshift-domain>:6443/healthz
* Added api-int.<cluster-openshift-domain>:6443:<infra_node_ip> to DNS cache
* Hostname api-int.<cluster-openshift-domain> was found in DNS cache
*   Trying 10.xx.xxx.xxx:6443...
* connect to 10.xx.xxx.xxx port 6443 failed: Connection refused
* Failed to connect to api-int.<cluster-openshift-domain> port 6443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to api-int.<cluster-openshift-domain>port 6443: Connection refused
  • Confirm if there is no LB configuration issue by adding api-int entry with master node ip in /etc/hosts on one of the masters and then run curl api-int:
# curl -vk https://api-int.<cluster-openshift-domain>:6443
*   Trying 10.xx.xxx.xx:6443...
* Connected to api-int.<cluster-openshift-domain> (10.xx.xxx.xxx) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
..
  • Verify if the Load Balancer is accessible(Curl to LB IP):
# curl -kv https://<LB_IP>:6443
*   Trying 10.15.xxx.xxx:6443...
* connect to 10.15.xxx.xxx port 6443 failed: Connection refused
* Failed to connect to 10.15.xxx.xxx port 6443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 10.15.xxx.xxx port 6443: Connection refused

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments