The IP addresses from the Ingress Load Balancer are changing frequently

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Service on AWS [ROSA]
    • 4.x
  • Red Hat OpenShift Dedicated on AWS [OSD]
    • 4.x

Issue

  • The IP addresses from the Ingress Load Balancer are changing frequently, which causes connectivity issues when trying to access the Openshift Web Console and Apps routes.
  • How to avoid Ingress Load Balancer IP to be changed?
  • I am using the IP addresses provided by the Ingress Load Balancer to allow traffic on my Firewall, and it is causing issues when the IP change.

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

  • For the AWS Elastic Load Balancing, as it frequently changes its IP addresses, it is recommended to use the provided Fully Qualified Domain Name - (FQDN) instead of the allocated IPs. For more details, please refer to the AWS Documentation.

  • If the IP addresses are still required to allow Firewall/Proxy's traffic, there is another alternative by using Custom Domains, where the Custom Domains Operator could be configured with loadBalancerType: NLB. For more details refer to the article Using NLB in OSD or ROSA.

  • Another point to be aware when working with OpenShift, especially with ROSA and OSD clusters on AWS is that, by default, the Default Ingress Controller uses the AWS Classic Load Balancer for routing the cluster's ingress traffic. As a standard behavior, the Load Balancer will assume all the cluster nodes on the available Instances, however, only the nodes which have the router pods will be considered InService. Example:

    $ oc get pods -n openshift-ingress -o wide
NAME                                    READY   STATUS    RESTARTS   AGE   IP            NODE                                              NOMINATED NODE   READINESS GATES
router-default-59769c4d76-4jrwm         1/1     Running   0          42h   10.X.X.X    ip-10-X-X-X.ap-southeast-1.compute.internal   <none>           <none>
router-default-59769c4d76-svwk2         1/1     Running   0          42h   10.X.X.X    ip-10-X-X-X.ap-southeast-1.compute.internal    <none>           <none>

    Important: These router pods may also be replaced at any time and scheduled in different Infra nodes. So, it is important not consider the IPs from the nodes as it may also change overtime, or perhaps, consider the IP range from the Machine CIDR, if required.

Root Cause

  • By default, the IP addresses for AWS Classic Load Balancers and Application Load Balancers change over time. Avoid using this information to statically configure your applications to point to these IP addresses.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments