Managing ingresses with the new ingress feature implemented on OpenShift release 4.14

Resolution

• Scenario 1: Changing the type of my default Ingress Controller (or additional ingresses)
• Scenario 2: Upgrading my cluster from release 4.13 to 4.14 with custom domains for applications

Scenario 1: Changing the type of my default Ingress Controller (or additional ingresses)

For this scenario, we are going to demonstrate how to switch the default Ingress Controller on ROSA and OSD clusters from a Network Load Balancer (NLB) to a Classic Load Balancer (CLB). Click the link for your cluster type to jump to the appropriate section:

• ROSA Clusters
• OSD Clusters

Please be aware that this process will be completed once the DNS propagation is done, which could take around ~15-45 minutes.

Note: On versions 4.14 and above, clusters are provisioned with a NLB on the default Ingress Controller.

Important: After your existing Custom Domain is deprecated, the underlying IngressController resource will have a different appsDomain to your external top-level domain. This is the intermediary domain that is then referred to via a CNAME in your external DNS records. This Ingress Controller was created when initially setting up your Custom Domain (on OpenShift Dedicated or on ROSA). The defaultCertificate that is linked to the Custom Domain, as well as the IngressController you will use upon deprecation, is valid for the hostname of your Custom Domain (and not the hostname of the appsDomain endpoint), in the same way as it was before deprecation.

Important: When rotating the certificate of your IngressController after the deprecation of your Custom Domain, follow the Openshift Container Platform guide for updating the default Ingress Certificate, using your custom Ingress Controller instead of the default one. You will need to create a new secret, or update the existing secret linked to your Ingress Controller's defaultCertificate in the openshift-ingress namespace.

Important: If you use ROSA or OSD version 4.13 or lower, you need to request Early Access to change your ingress controller. If you follow this procedure without being granted Early Access to this feature, you will see an error similar to the following:

    E: Updating Load Balancer Type is not supported for STS clusters on legacy ingress support

ROSA Clusters

Prerequisites:

• rosa cli
• oc cli
• Access to AWS console for validation

Procedure:

1. List existing ingresses for the cluster:

   $ rosa list ingresses --cluster=<clusterName|clusterID>
   

   Example:

   $ rosa list ingresses --cluster=f9f6284f-d0ef-4751-bbcc-xxxxxxxxxxxx
   ID    APPLICATION ROUTER                                      PRIVATE  DEFAULT  ROUTE SELECTORS  LB-TYPE  EXCLUDED NAMESPACE  WILDCARD POLICY      NAMESPACE OWNERSHIP  HOSTNAME  TLS SECRET REF
   w1z4  https://apps.<clusterName>.w1ju.p1.openshiftapps.com    no       yes                       nlb                          WildcardsDisallowed  Strict                         
   

2. Take note of the ID provided on the previous step and update the ingress type:

   $ rosa edit ingress --cluster=f9f6284f-d0ef-4751-bbcc-xxxxxxxxxxxx --lb-type=classic w1z4
   I: Updated ingress 'w1z4' on cluster 'f9f6284f-d0ef-4751-bbcc-xxxxxxxxxxxx'
   

Please refer to Checking the cluster status after changing the Load Balancer type for the remaining steps.

OSD Clusters

Pre-requisites:

• ocm cli
• oc cli
• Access to AWS console for validation

Procedure:

1. List existing ingresses for the cluster:

   $ ocm list ingresses --cluster=<clusterName|clusterID>
   

   Example:

   $ ocm list ingresses --cluster=d38eb89e-fc60-4b47-af83-xxxxxxxxxxxx
   ID   API ENDPOINT                                            PRIVATE 
   api  https://api.<clusterName>.r3yh.p1.openshiftapps.com:6443  external
   
   ID    APPLICATION ROUTER                                  PRIVATE   DEFAULT  ROUTE SELECTORS
   f2z3  https://apps.<clusterName>.r3yh.p1.openshiftapps.com  external  true                    
   

2. Take note of the ID provided for the "APPLICATION ROUTER" and update the ingress type:

   $ ocm edit ingress --cluster=d38eb89e-fc60-4b47-af83-xxxxxxxxxxxx --lb-type=classic f2z3
   <no output>
   

Please refer to Checking the cluster status after changing the Load Balancer type for the remaining steps.

Checking the cluster status after changing the Load Balancer type

The following steps can be used for both ROSA/OSD clusters:

1. Once the steps to change the Load Balancer type were successful, you will notice that the router pods will start to be rolled out:

   $ oc get pods -n openshift-ingress
   NAME                              READY   STATUS        RESTARTS   AGE
   router-default-68dbf7d64-5gfzs    1/1     Running       0          37s
   router-default-68dbf7d64-kkshr    1/1     Running       0          37s
   router-default-6b7548fdf4-nlzln   1/1     Terminating   0          101m
   router-default-6b7548fdf4-t7cll   0/1     Terminating   0          103m
   

2. Also, during the process, until you have the AWS load balance active and the new DNS propagated, you will notice that the cluster operators authentication and console will show as Degraded:

   $ oc get co authentication console
   NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   authentication   4.14.6    False       False         True       3m20s   OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.<clusterName>.w1ju.p1.openshiftapps.com/healthz": dial tcp: lookup oauth-openshift.apps.<clusterName>.w1ju.p1.openshiftapps.com on 172.X.X.10:53: no such host (this is likely result of malfunctioning DNS server)
   console          4.14.6    False       False         False      3m25s   RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.<clusterName>.w1ju.p1.openshiftapps.com): Get "https://console-openshift-console.apps.<clusterName>.w1ju.p1.openshiftapps.com": dial tcp: lookup console-openshift-console.apps.<clusterName>.w1ju.p1.openshiftapps.com on 172.X.X.10:53: no such host
   

3. Checking your AWS load balancers, you can see the change on the load balancer type:

   From:
   

   To:
   

4. You can also check the services on openshift-ingress namespace and verify if it matches with the load balancer on AWS:

   $ oc get svc -n openshift-ingress router-default
   

Scenario 2: Upgrading my cluster from release 4.13 to 4.14 with custom domains for applications

For this scenario, we are going to perform an upgrade of a ROSA cluster from 4.13 to 4.14 and see what will be the behavior of the ingresses and the existing customdomain. This can also be considered for OSD clusters, it is just required to swap rosa cli with ocm cli.

Pre-requisites:

• rosa cli
• oc cli
• Access to AWS console for validation

Procedure:

1. List existing ingresses for the cluster:

   $ rosa list ingresses --cluster=<clusterName|clusterID>
   ID    APPLICATION ROUTER                                 PRIVATE  DEFAULT  ROUTE SELECTORS  LB-TYPE  EXCLUDED NAMESPACE  WILDCARD POLICY  NAMESPACE OWNERSHIP  HOSTNAME  TLS SECRET REF
   i7m0  https://apps.<clusterName>.rkj6.s1.devshift.org  no       yes                       classic                                                                      
   

2. List existing customdomains:

   $ oc get customdomains
   NAME         ENDPOINT                                                 DOMAIN           STATUS
   testdomain   nmkpsf.testdomain.<clusterName>.rkj6.s1.devshift.org     testdomain.org   Ready
   

3. List existing ingresscontrollers:

   $ oc get ingresscontrollers -n openshift-ingress-operator
   NAME         AGE
   default      77m
   testdomain   12m
   

4. Check the services on openshift-ingress namespace:

   $ oc get svc -n openshift-ingress
   

5. Until now, the existing customdomains are up and running as shown below:

   $ oc get customdomain testdomain -o yaml
   apiVersion: v1
   items:
   - apiVersion: managed.openshift.io/v1alpha1
     kind: CustomDomain
     metadata:
       name: testdomain
     status:
       conditions:
       - lastProbeTime: "2024-01-19T06:00:09Z"
         lastTransitionTime: "2024-01-19T06:00:09Z"
         message: Custom Apps Domain (testdomain.org) Is Ready
         reason: Ready
         status: "True"
         type: Ready
       dnsRecord: '*.testdomain.<clusterName>iftw.s1.devshift.org.'
       endpoint: itjdid.testdomain.<clusterName>.iftw.s1.devshift.org
       scope: ""
       state: Ready
   

6. Proceed with cluster upgrade by following the official documentation. The following events are expected when running the upgrade command:

   ? Are you sure you want to upgrade cluster to version '4.14.7'? Yes
   W: Missing required acknowledgements to schedule upgrade. 
   
   ? Read the below description and acknowledge to proceed with upgrade
     - Description: Upgrade to 4.14 may affect additional Ingress behaviour.
       Warning:     In version 4.14 of OSD/ROSA, we are changing how we manage Ingress Controllers. If you are using additional non-default Ingresses, or Custom Domains, your cluster may be affected.
       URL:         https://access.redhat.com/node/7028653
   
   ? I acknowledge (Y/n)
   
   I: Gate 4b6136f6-46c2-11ee-ae96-0a580a821aab acknowledged
   ? Read the below description and acknowledge to proceed with upgrade
     - Description: OpenShift removes several Kubernetes APIs in OpenShift 4.14.
       Warning:     To prevent an outage on your cluster, review any APIs in use that will be removed, and migrate them to the appropriate new API version. Failure to evaluate and migrate components affected by this update can cause some types of workloads to stop functioning.
       URL:         https://access.redhat.com/articles/6958395
   
   ? I acknowledge (Y/n) 
   I: Gate 9f2d78a1-57d0-11ee-949e-0a580a800925 acknowledged
   I: Upgrade successfully scheduled for cluster '<clusterName>'
   

7. Once the upgrade is finished, if you list the customdomains, you will continue seeing them as a resource, however, it does not have any function anymore, its condition will show as Deprecated. This basically tells the user that the custom domain object is still there, but any changes to it will not be reconciled:

   $ oc get customdomain testdomain -o yaml
   apiVersion: v1
   items:
   - apiVersion: managed.openshift.io/v1alpha1
     kind: CustomDomain
     metadata:
       name: testdomain
     status:
       conditions:
       - lastProbeTime: "2024-01-19T08:01:02Z"
     lastTransitionTime: "2024-01-19T08:01:02Z"
     message: Due to the deprecation of the custom domains operator on OSD/ROSA version 4.13 and above, this CustomDomain no longer manages an IngressController.
     reason: Deprecated
     status: "True"
     type: Deprecated
       dnsRecord: '*.testdomain.<clusterName>iftw.s1.devshift.org.'
       endpoint: itjdid.testdomain.<clusterName>.iftw.s1.devshift.org
       scope: ""
   state: NotReady
   

8. Checking the ingresscontrollers again, they are still there. This is because the customdomains use ingresscontroller resources under the hood, so they will still stay as an ingresscontroller in the openshift-ingress-operator namespace. However, this ingress will no longer be managed by the Custom Domains Operator:

   $ oc get ingresscontrollers -n openshift-ingress-operator
   NAME         AGE
   default      3h12m
   testdomain   147m
   

9. Listing the existing ingresses with rosa cli, it will still show the default ingress:

   $ rosa list ingresses --cluster=<clusterName|clusterID>
   ID    APPLICATION ROUTER                                 PRIVATE  DEFAULT  ROUTE SELECTORS  LB-TYPE  EXCLUDED NAMESPACE  WILDCARD POLICY  NAMESPACE OWNERSHIP  HOSTNAME  TLS SECRET REF
   i7m0  https://apps.<clusterName>.rkj6.s1.devshift.org  no       yes                       classic                                                                      
   

10. But checking in the AWS Console, the load balancer will still be available and running.

11. If you need to change a non-default ingresscontroller, you need to change the spec.endpointPublishingStrategy using the OC CLI. For more information, see the OpenShift Container Platform documentation:

• Understanding the Ingress Operator
• Configuring the Ingress Controller endpoint publishing strategy.