How Ingress router default certificate is managed in Azure Red Hat OpenShift 4

Solution Verified - Updated -

Environment

  • Red Hat OpenShift on Azure (ARO)
    • 4

Issue

  • Ingress router default certificate is about to expire
  • How to confirm which certificates are used for Ingress router default?
  • How to identify which CA is signing default certificates?
  • Which code block is used for auto renewing certificates?

Resolution

  • In Azure Red Hat OpenShift (ARO), when no custom domain certificate has been applied to the cluster, the certificate renewals for the default ingress controllers are managed by Red Hat as part of the ARO service and is auto renewed. This is managed, by the tls.go module by requesting a certificate from one of the Azure Certificate Authorities. The certificate is then stored in the clusters Azure Keyvault where it is then recalled and added as a cluster secret.

  • This means that Azure Red Hat OpenShift 4 (ARO) does not require the certificate authority which is provided in the the cert-utils operator.

  • If custom domain certificates have been applied to the cluster, then the cluster administrators will need to maintain the certificate.

Diagnostic Steps

  1. Confirm the default certificate is mounted in the router-default deployment in openshift-ingress namespace

    $ oc get deployment/router-default -o yaml -n openshift-ingress| grep -A10 volumes

  volumes:
  - name: default-certificate
    secret:
      defaultMode: 420
      secretName: xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-ingress   <------ cert is mounted in the deployment/router-default
  - configMap:
      defaultMode: 420
      items:
      - key: service-ca.crt
        path: service-ca.crt
      name: service-ca-bundle
............output omitted.................

    $ oc get secrets -n openshift-ingress | grep -i ingress

xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-ingress   kubernetes.io/tls

  2. Check the default router certificate to confirm aroapp.io domain is used. Confirm the certificate expiration date and CA signing the certificate. Azure is the expected singing CA.

    $ echo -e "NAMESPACE\tNAME\tEXPIRY" && oc get secrets -n openshift-ingress  -o go-template='{{range .items}}{{if eq .type "kubernetes.io/tls"}}{{.metadata.namespace}}{{" "}}{{.metadata.name}}{{" "}}{{index .data "tls.crt"}}{{"\n"}}{{end}}{{end}}'  | while read namespace name cert; do echo -en "$namespace\t$name\t"; echo $cert | base64 -d | openssl x509 -noout -text ; done


NAMESPACE    NAME    EXPIRY
openshift-ingress    xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-ingress  Certificate:

    Signature Algorithm: sha384WithRSAEncryption
    Issuer: C = US, O = Microsoft Corporation, CN = Microsoft Azure TLS Issuing CA 05
    Validity
        Not Before: Jul  6 05:11:18 2023 GMT      
        Not After : Jun 27 23:59:59 2024 GMT                  <----------------- Expiration date
    Subject: C = US, ST = WA, L = Redmond, O = Microsoft Corporation, 
        CN = *.apps.u1kgiy5t.centralindia.aroapp.io           <------- Azure aroapp.io domain is used
    Subject Public Key Info:
        Public Key Algorithm: rsaEncryption
            RSA Public-Key: (2048 bit)
            Modulus:
............output omitted.................

  3. Check if openshift-ingress-operator has self-signed router-ca certificate. This is only one place where it should be used.

    $ echo -e "NAMESPACE\tNAME\tEXPIRY" && oc get secrets -n openshift-ingress-operator  -o go-template='{{range .iems}}{{if eq .type "kubernetes.io/tls"}}{{.metadata.namespace}}{{" "}}{{.metadata.name}}{{" "}}{{index .data "tls.crt"}}{{"\n"}}{{end}}{{end}}'  | while read namespace name cert; do echo -en "$namespace\t$name\t"; echo $cert | base64 -d | openssl x509 -noout -text ; done

openshift-ingress-operator   router-ca   Certificate:
Data:
    Version: 3 (0x2)
    Serial Number: 1 (0x1)
    Signature Algorithm: sha256WithRSAEncryption
    Issuer: CN = ingress-operator@xxxxxx
    Validity
        Not Before: Jul  6 05:30:09 2023 GMT
        Not After : Jul  5 05:30:10 2025 GMT
    Subject: CN = ingress-operator@xxxxxx    <--- the only one Ingress Controller cert is not signed by the Azure CA
    Subject Public Key Info:
        Public Key Algorithm: rsaEncryption
            RSA Public-Key: (2048 bit)
            Modulus:
............output omitted.................

  4. Check and make sure that no other secrets/certificates are using certificate issued by ingress-operator@xxxxxx

    $ echo -e "NAMESPACE\tNAME\tEXPIRY" && oc get secrets -A  -o go-template='{{range .items}}{{if eq .type "kubernetes.io/tls"}}{{.metadat.namespace}}{{" "}}{{.metadata.name}}{{" "}}{{index .data "tls.crt"}}{{"\n"}}{{end}}{{end}}'  | while read namespace name cert; do echo -en "$namespace\t$name\t"; echo $cert | base64 -d | openssl x509 -noout -text ; done | grep -i -A10 -B10 'ingress-operator@'

openshift-ingress-operator   router-ca   Certificate:
Data:
    Version: 3 (0x2)
    Serial Number: 1 (0x1)
    Signature Algorithm: sha256WithRSAEncryption
    Issuer: CN = ingress-operator@xxxxxx
    Validity
        Not Before: Jul  6 05:30:09 2023 GMT
        Not After : Jul  5 05:30:10 2025 GMT
    Subject: CN = ingress-operator@xxxxxx
    Subject Public Key Info:
        Public Key Algorithm: rsaEncryption
            RSA Public-Key: (2048 bit)
            Modulus:
............output omitted.................

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments