Chapter 5. Configuring certificates

5.1. Replacing the default ingress certificate

5.1.1. Understanding the default ingress certificate

By default OpenShift Container Platform uses the Ingress Operator to create an internal CA and issue a wildcard certificate that is valid for applications under the .apps sub-domain. Both the web console and CLI use this certificate as well.

The internal infrastructure CA certificates are self-signed. While this process might be perceived as bad practice by some security or PKI teams, any risk here is minimal. The only clients that implicitly trust these certificates are other components within the cluster. Replacing the default wildcard certificate with one that is issued by a public CA already included in the CA bundle as provided by the container userspace allows external clients to connect securely to applications running under the .apps sub-domain.

5.1.2. Replacing the default ingress certificate

You can replace the default ingress certificate for all applications under the .apps subdomain. After you replace the certificate, all applications, including the web console and CLI, will have encryption provided by specified certificate.

Prerequisites

  • You must have a wildcard certificate and its private key, both in the PEM format, for use.
  • The certificate must have a subjectAltName extension of *.apps.<clustername>.<domain>.

Procedure

  1. Create a ConfigMap that includes the certificate authority used to signed the new certificate:

    $ oc create configmap custom-ca \
         --from-file=ca-bundle.crt=</path/to/example-ca.crt> \1
         -n openshift-config
    1
    </path/to/example-ca.crt> is the path to the certificate authority file on your local file system.
  2. Update the cluster-wide proxy configuration with the newly created ConfigMap:

    $ oc patch proxy/cluster \
         --type=merge \
         --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}'
  3. Create a secret that contains the wildcard certificate and key:

    $ oc create secret tls <certificate> \1
         --cert=</path/to/cert.crt> \2
         --key=</path/to/cert.key> \3
         -n openshift-ingress
    1
    <certificate> is the name of the secret that will contain the certificate and private key.
    2
    </path/to/cert.crt> is the path to the certificate on your local file system.
    3
    </path/to/cert.key> is the path to the private key associated with this certificate.
  4. Update the Ingress Controller configuration with the newly created secret:

    $ oc patch ingresscontroller.operator default \
         --type=merge -p \
         '{"spec":{"defaultCertificate": {"name": "<certificate>"}}}' \1
         -n openshift-ingress-operator
    1
    Replace <certificate> with the name used for the secret in the previous step.

5.2. Adding API server certificates

The default API server certificate is issued by an internal OpenShift Container Platform cluster CA. Clients outside of the cluster will not be able to verify the API server’s certificate by default. This certificate can be replaced by one that is issued by a CA that clients trust.

5.2.1. Add an API server named certificate

The default API server certificate is issued by an internal OpenShift Container Platform cluster CA. You can add additional certificates to the API server to send based on the client’s requested URL, such as when a reverse proxy or load balancer is used.

Prerequisites

  • You must have the certificate and key, in the PEM format, for the client’s URL.
  • The certificate must be issued for the URL used by the client to reach the API server.
  • The certificate must have the subjectAltName extension for the URL.
  • If a certificate chain is required to certify the server certificate, then the certificate chain must be appended to the server certificate. Certificate files must be Base64 PEM-encoded and typically have a .crt or .pem extension. For example:

    $ cat server_cert.pem int2ca_cert.pem int1ca_cert.pem rootca_cert.pem>combined_cert.pem

    When combining certificates, the order of the certificates is important. Each following certificate must directly certify the certificate preceding it, for example:

    1. OpenShift Container Platform master host server certificate.
    2. Intermediate CA certificate that certifies the server certificate.
    3. Root CA certificate that certifies the intermediate CA certificate.
Warning

Do not provide a named certificate for the internal load balancer (host name api-int.<cluster_name>.<base_domain>). Doing so will leave your cluster in a degraded state.

Procedure

  1. Create a secret that contains the certificate and key in the openshift-config namespace.

    $ oc create secret tls <certificate> \1
         --cert=</path/to/cert.crt> \2
         --key=</path/to/cert.key> \3
         -n openshift-config
    1
    <certificate> is the name of the secret that will contain the certificate.
    2
    </path/to/cert.crt> is the path to the certificate on your local file system.
    3
    </path/to/cert.key> is the path to the private key associated with this certificate.
  2. Update the API server to reference the created secret.

    $ oc patch apiserver cluster \
         --type=merge -p \
         '{"spec":{"servingCerts": {"namedCertificates":
         [{"names": ["<hostname>"], 1
         "servingCertificate": {"name": "<certificate>"}}]}}}' 2
    1
    Replace <hostname> with the hostname that the API server should provide the certificate for.
    2
    Replace <certificate> with the name used for the secret in the previous step.
  3. Examine the apiserver/cluster object and confirm the secret is now referenced.

    $ oc get apiserver cluster -o yaml
    ...
    spec:
      servingCerts:
        namedCertificates:
        - names:
          - <hostname>
          servingCertificate:
            name: <certificate>
    ...

5.3. Securing service traffic using service serving certificate secrets

5.3.1. Understanding service serving certificates

Service serving certificates are intended to support complex middleware applications that require encryption. These certificates are issued as TLS web server certificates.

The service-ca controller uses the x509.SHA256WithRSA signature algorithm to generate service certificates.

The generated certificate and key are in PEM format, stored in tls.crt and tls.key respectively, within a created secret. The certificate and key are automatically replaced when they get close to expiration. The service CA certificate, which signs the service certificates, is only valid for one year after OpenShift Container Platform is installed.

5.3.2. Add a service certificate

To secure communication to your service, generate a signed serving certificate and key pair into a secret in the same namespace as the service.

Important

The generated certificate is only valid for the internal service DNS name <service.name>.<service.namespace>.svc, and are only valid for internal communications.

Prerequisites:

  • You must have a service defined.

Procedure

  1. Annotate the service with service.beta.openshift.io/serving-cert-secret-name.

    $ oc annotate service <service-name> \1
         service.beta.openshift.io/serving-cert-secret-name=<secret-name> 2
    1
    Replace <service-name> with the name of the service to secure.
    2
    <secret-name> will be the name of the generated secret containing the certificate and key pair. For convenience, it is recommended that this be the same as <service-name>.

    For instance, use the following command to annotate the service foo:

    $ oc annotate service foo service.beta.openshift.io/serving-cert-secret-name=foo
  2. Examine the service to confirm the annotations are present.

    $ oc describe service <service-name>
    ...
    Annotations:              service.beta.openshift.io/serving-cert-secret-name: <service-name>
                              service.beta.openshift.io/serving-cert-signed-by: openshift-service-serving-signer@1556850837
    ...
  3. After the cluster generates a secret for your service, your PodSpec can mount it, and the Pod will run after it becomes available.

5.3.3. Add a service certificate to a ConfigMap

A Pod can access the service CA certificate by mounting a ConfigMap that is annotated with service.beta.openshift.io/inject-cabundle=true. Once annotated, the cluster automatically injects the service CA certificate into the service-ca.crt key on the ConfigMap. Access to this CA certificate allows TLS clients to verify connections to services using service serving certificates.

Important

After adding this annotation to a ConfigMap all existing data in it is deleted. It is recommended to use a separate ConfigMap to contain the service-ca.crt, instead of using the same ConfigMap that stores your Pod’s configuration.

Procedure

  1. Annotate the ConfigMap with service.beta.openshift.io/inject-cabundle=true.

    $ oc annotate configmap <configmap-name> \1
         service.beta.openshift.io/inject-cabundle=true
    1
    Replace <configmap-name> with the name of the ConfigMap to annotate.
    Note

    Explicitly referencing the service-ca.crt key in a volumeMount will prevent a Pod from starting until the ConfigMap has been injected with the CA bundle.

    For instance, to annotate the ConfigMap foo the following command would be used:

    $ oc annotate configmap foo service.beta.openshift.io/inject-cabundle=true
  2. View the ConfigMap to ensure the certificate has been generated. This appears as a service-ca.crt in the YAML output.

    $ oc get configmap <configmap-name> -o yaml
    apiVersion: v1
    data:
      service-ca.crt: |
        -----BEGIN CERTIFICATE-----
    ...

5.3.4. Manually rotate the generated service certificate

You can rotate the service certificate by deleting the associated secret. Deleting the secret results in a new one being automatically created, resulting in a new certificate.

Prerequisites

  • A secret containing the certificate and key pair must have been generated for the service.

Procedure

  1. Examine the service to determine the secret containing the certificate. This is found in the serving-cert-secret-name annotation, as seen below.

    $ oc describe service <service-name>
    ...
    service.beta.openshift.io/serving-cert-secret-name: <secret>
    ...
  2. Delete the generated secret for the service. This process will automatically recreate the secret.

    $ oc delete secret <secret> 1
    1
    Replace <secret> with the name of the secret from the previous step.
  3. Confirm that the certificate has been recreated by obtaining the new secret and examining the AGE.

    $ oc get secret <service-name>
    
    NAME              TYPE                DATA   AGE
    <service.name>    kubernetes.io/tls   2      1s

5.3.5. Manually rotate the service CA certificate

The service CA is valid for one year after OpenShift Container Platform is installed. Follow these steps to manually refresh the service CA before the expiration date.

Prerequisites

  • You must be logged in as a cluster admin.

Procedure

  1. View the expiration date of the current service CA certificate by using the following command.

    $ oc get secrets/signing-key -n openshift-service-ca \
         -o template='{{index .data "tls.crt"}}' \
         | base64 -d \
         | openssl x509 -noout -enddate
  2. Manually rotate the service CA. This process generates a new service CA which will be used to sign the new service certificates.

    $ oc delete secret/signing-key -n openshift-service-ca
  3. To apply the new certificates to all services, restart all the Pods in your cluster. This command ensures that all services use the updated certificates.

    $ for I in $(oc get ns -o jsonpath='{range .items[*]} {.metadata.name}{"\n"} {end}'); \
          do oc delete pods --all -n $I; \
          sleep 1; \
          done
    Warning

    This command will cause a service interruption, as it goes through and deletes every running pod in every namespace. These pods will automatically restart after they are deleted.