Chapter 1. Risk and compliance

Manage your security of Red Hat Advanced Cluster Management for Kubernetes components. Govern your cluster with defined policies and processes to identify and minimize risks. Use policies to define rules and set controls.

Prerequisite: You must configure authentication service requirements for Red Hat Advanced Cluster Management for Kubernetes. See Access control for more information.

Review the following topics to learn more about securing your cluster:

1.1. Certificates

Various certificates are created and used throughout Red Hat Advanced Cluster Management for Kubernetes.

You can bring your own certificates. You must create a Kubernetes TLS Secret for your certificate. After you create your certificates, you can replace certain certificates that are created by the Red Hat Advanced Cluster Management installer.

Required access: Cluster administrator or team administrator.

Note: Replacing certificates is supported only on native Red Hat Advanced Cluster Management installations.

All certificates required by services that run on Red Hat Advanced Cluster Management are created during the installation of Red Hat Advanced Cluster Management. Certificates are created and managed by the OpenShift Service Serving Certificates service.

You can also rotate the OpenShift Service Serving certificates. For more information, follow the OpenShift documentation to Manually rotate the generated service certificate and Manually rotate the service CA certificate. After the rotation is complete, apply the new certificates to all of the services with the following command:

oc -n open-cluster-management delete pod -l chart=management-ingress

The related pods in your cluster restart automatically.

Continue reading to learn more about certificate management:

Red Hat Advanced Cluster Management hub cluster certificates

Red Hat Advanced Cluster Management component certificates

Red Hat Advanced Cluster Management managed certificates

Third-party certificates

1.1.1. Red Hat Advanced Cluster Management hub cluster certificates

1.1.1.1. Observability certificates

After Red Hat Advanced Cluster Management is installed, observability certificates are created and used by the observability components, to provide mutual TLS on the traffic between the hub cluster and managed cluster. The Kubernetes secrets that are associated with the observability certificates.

The open-cluster-management-observability namespace contain the following certificates:

  • observability-server-ca-certs: Has the CA certificate to sign server-side certificates
  • observability-client-ca-certs: Has the CA certificate to sign client-side certificates
  • observability-server-certs: Has the server certificate used by the observability-observatorium-api deployment
  • observability-grafana-certs: Has the client certificate used by the observability-rbac-query-proxy deployment

The open-cluster-management-addon-observability namespace contain the following certificates on managed clusters:

  • observability-managed-cluster-certs: Has the same server CA certificate as observability-server-ca-certs in the hub server
  • observability-controller-open-cluster-management.io-observability-signer-client-cert: Has the client certificate used by the metrics-collector-deployment

The CA certificates are valid for five years and other certificates are valid for one year. All observability certificates are automatically refreshed upon expiration.

View the following list to understand the effects when certificates are automatically renewed:

  • Non-CA certificates are renewed automatically when the remaining valid time is no more than 73 days. After the certificate is renewed, the pods in the related deployments restart automatically to use the renewed certificates.
  • CA certificates are renewed automatically when the remaining valid time is no more than one year. After the certificate is renewed, the old CA is not deleted but co-exist with the renewed ones. Both old and renewed certificates are used by related deployments, and continue to work. The old CA certificates are deleted when they expire.
  • When a certificate is renewed, the traffic between the hub cluster and managed cluster is not interrupted.

1.1.1.2. Bring Your Own (BYO) observability certificate authority (CA) certificates

If you do not want to use the default observability CA certificates generated by Red Hat Advanced Cluster Management, you can choose to use the BYO observability CA certificates before you enable observability.

1.1.1.2.1. OpenSSL commands to generate CA certificate

Observability requires two CA certificates; one is for the server-side and the other is for the client-side.

  • Generate your CA RSA private keys with the following commands:

    openssl genrsa -out serverCAKey.pem 2048
    
    openssl genrsa -out clientCAKey.pem 2048
  • Generate the self-signed CA certificates using the private keys. Run the following commands:

    openssl req -x509 -sha256 -new -nodes -key serverCAKey.pem -days 1825 -out serverCACert.pem
    
    openssl req -x509 -sha256 -new -nodes -key clientCAKey.pem -days 1825 -out clientCACert.pem
1.1.1.2.2. Create the secrets associated with the BYO observability CA certificates

Complete the following steps to create the secrets:

  1. Create the observability-server-ca-certs secret by using your certificate and private key. Run the following command:

    oc -n open-cluster-management-observability create secret tls observability-server-ca-certs --cert ./serverCACert.pem --key ./serverCAKey.pem
  2. Create the observability-client-ca-certs secret by using your certiicate and private key. Run the following command:

    oc -n open-cluster-management-observability create secret tls observability-client-ca-certs --cert ./clientCACert.pem --key ./clientCAKey.pem
1.1.1.2.3. Replacing certificates for alertmanager route

You can replace alertmanager certificates by updating the alertmanager route, if you do not want to use the OpenShift default ingress certificate. Complete the following steps:

  1. Examine the observability certificate with the following command:

    openssl x509  -noout -text -in ./observaility.crt
  2. Change the common name (CN) on the certificate to alertmanager.
  3. Change the SAN in the csr.cnf configuration file with the hostname for your alertmanager route.
  4. Create the two following secrets in the open-cluster-management-observability namespace. Run the following command:

    oc -n open-cluster-management-observability create secret tls alertmanager-byo-ca --cert ./ca.crt --key ./ca.key
    
    oc -n open-cluster-management-observability create secret tls alertmanager-byo-cert --cert ./ingress.crt --key ./ingress.key

For more information, see OpenSSL commands for generating a certificate. If you want to restore the default self-signed certificate for alertmanager route, see Restore the default self-signed certificate for management ingress to delete the two secrets in the open-cluster-management-observability namespace.

1.1.2. Red Hat Advanced Cluster Management component certificates

1.1.2.1. List hub cluster managed certificates

You can view a list of hub cluster managed certificates that use OpenShift Service Serving Certificates service internally. Run the following command to list the certificates:

oc get secret -n open-cluster-management -o custom-columns=Name:.metadata.name,Expiration:.metadata.annotations.service\\.beta\\.openshift\\.io/expiry | grep -v '<none>'

Note: If observability is enabled, there are additional namespaces where certificates are created.

1.1.2.2. Refresh hub cluster managed certificates

You can refresh a hub cluster managed certificate by running the command in the List hub cluster managed certificates section. When you identify the certificate that you need to refresh, delete the secret that is associated with the certificate. For example, you can delete a secret by running the following command:

oc delete secret grc-0c925-grc-secrets -n open-cluster-management

Note: After you delete the secret, a new one is created. However, you must restart pods that use the secret manually so they can begin to use the new certificate.

1.1.2.3. Refresh a OpenShift Container Platform managed certificate

You can refresh OpenShift Container Platform managed certificates, which are certificates that are used by Red Hat Advanced Cluster Management webhooks and the proxy server.

Complete the following steps to refresh OpenShift Container Platform managed certificates:

  1. Delete the secret that is associated with the OpenShift Container Platform managed certificate by running the following command:

    oc delete secret -n open-cluster-management ocm-webhook-secret

    Note: Some services might not have a secret that needs to be deleted.

  2. Restart the services that are associated with the OpenShift Container Platform managed certificate(s) by running the following command:

    oc delete po -n open-cluster-management ocm-webhook-679444669c-5cg76

    Important: There are replicas of many services; each service must be restarted.

View the following table for a summarized list of the pods that contain certificates and whether a secret needs to be deleted prior to restarting the pod:

Table 1.1. Pods that contain OpenShift Container Platform managed certificates

Service nameNamespaceSample pod nameSecret name (if applicable)

channels-apps-open-cluster-management-webhook-svc

open-cluster-management

multicluster-operators-application-8c446664c-5lbfk

-

multicluster-operators-application-svc

open-cluster-management

multicluster-operators-application-8c446664c-5lbfk

-

multiclusterhub-operator-webhook

open-cluster-management

multiclusterhub-operator-bfd948595-mnhjc

-

ocm-webhook

open-cluster-management

ocm-webhook-679444669c-5cg76

ocm-webhook-secret

cluster-manager-registration-webhook

open-cluster-management-hub

cluster-manager-registration-webhook-fb7b99c-d8wfc

registration-webhook-serving-cert

cluster-manager-work-webhook

open-cluster-management-hub

cluster-manager-work-webhook-89b8d7fc-f4pv8

work-webhook-serving-cert

1.1.3. Red Hat Advanced Cluster Management managed certificates

1.1.3.1. Channel certificates

CA certificates can be associated with Git channel that are a part of the Red Hat Advanced Cluster Management application management. See Using custom CA certificates for a secure HTTPS connection for more details.

Helm channels allow you to disable certificate validation. Helm channels where certificate validation is disabled, must be configured in development environments. Disabling certificate validation introduces security risks.

1.1.3.2. Managed cluster certificates

Certificates are used to authenticate managed clusters with the hub. Therefore, it is important to be aware of troubleshooting scenarios associated with these certificates. View Troubleshooting imported clusters offline after certificate change for more details.

The managed cluster certificates are refreshed automatically.

1.1.4. Third-party certificates

1.1.4.1. Rotating the gatekeeper webhook certificate

Complete the following steps to rotate the gatekeeper webhook certificate:

  1. Edit the secret that contains the certificate with the following command:

    oc edit secret -n openshift-gatekeeper-system gatekeeper-webhook-server-cert
  2. Delete the following content in the data section: ca.crt, ca.key, tls.crt`, and tls.key.
  3. Restart the gatekeeper webhook service by deleting the gatekeeper-controller-manager pods with the following command:

    oc delete po -n openshift-gatekeeper-system -l control-plane=controller-manager

The gatekeeper webhook certificate is rotated.

1.1.4.2. Rotating the integrity shield webhook certificate (Technology preview)

Complete the following steps to rotate the integrity shield webhook certificate:

  1. Edit the IntegrityShield custom resource and add the integrity-shield-operator-system namespace to the excluded list of namespaces in the inScopeNamespaceSelector setting. Run the following command to edit the resource:

    oc edit integrityshield integrity-shield-server -n integrity-shield-operator-system
  2. Delete the secret that contains the integrity shield certificate by running the following command:

    oc delete secret -n integrity-shield-operator-system ishield-server-tls
  3. Delete the operator so that the secret is recreated. Be sure that the operator pod name matches the pod name on your system. Run the following command:

    oc delete po -n integrity-shield-operator-system integrity-shield-operator-controller-manager-64549569f8-v4pz6
  4. Delete the integrity shield server pod to begin using the new certificate with the following command:

    oc delete po -n integrity-shield-operator-system integrity-shield-server-5fbdfbbbd4-bbfbz

Use the certificate policy controller to create and manage certificate policies on managed clusters. See Policy controllers to learn more about controllers. Return to the Risk and compliance page for more information.

1.2. Replacing the management ingress certificates

You can replace management ingress certificates by updating the Red Hat Advanced Cluster Management for Kubernetes route if you do not want to use the OpenShift default ingress certificate.

1.2.1. Prerequisites to replace management ingress certificate

Prepare and have your management-ingress certificates and private keys ready. If needed, you can generate a TLS certificate by using OpenSSL. Set the common name parameter (CN) on the certificate to manangement-ingress. If you are generating the certificate, include the following settings:

  • Include the route name for Red Hat Advanced Cluster Management for Kubernetes as the domain name in your certificate Subject Alternative Name (SAN) list.

    Receive the route name by running the following command:

    oc get route -n open-cluster-management

    You might receive the following response:

    multicloud-console.apps.grchub2.dev08.red-chesterfield.com

1.2.1.1. Example configuration file for generating a certificate

The following example configuration file and OpenSSL commands provide an example for how to generate a TLS certificate by using OpenSSL. View the following csr.cnf configuration file, which defines the configuration settings for generating certificates with OpenSSL.

[ req ]               # Main settings
default_bits = 2048       # Default key size in bits.
prompt = no               # Disables prompting for certificate values so the configuration file values are used.
default_md = sha256       # Specifies the digest algorithm.
req_extensions = req_ext  # Specifies the configuration file section that includes any extensions.
distinguished_name = dn   # Specifies the section that includes the distinguished name information.

[ dn ]               # Distinguished name settings
C = US                    # Country
ST = North Carolina             # State or province
L = Raleigh                # Locality
O = Red Hat Open Shift     # Organization
OU = Red Hat Advanced Container Management        # Organizational unit
CN = management-ingress  # Common name.

[ req_ext ]          # Extensions
subjectAltName = @alt_names # Subject alternative names

[ alt_names ]        # Subject alternative names
DNS.1 = multicloud-console.apps.grchub2.dev08.red-chesterfield.com

[ v3_ext ]          # x509v3 extensions
authorityKeyIdentifier=keyid,issuer:always  # Specifies the public key that corresponds to the private key that is used to sign a certificate.
basicConstraints=CA:FALSE                   # Indicates whether the certificate is a CA certificate during the certificate chain verification process.
#keyUsage=keyEncipherment,dataEncipherment   # Defines the purpose of the key that is contained in the certificate.
extendedKeyUsage=serverAuth                 # Defines the purposes for which the public key can be used.
subjectAltName=@alt_names                   # Identifies the subject alternative names for the identify that is bound to the public key by the CA.

Note: Be sure to update the SAN labeled, DNS.1 with the correct hostname for your management ingress.

1.2.1.2. OpenSSL commands for generating a certificate

The following OpenSSL commands are used with the preceding configuration file to generate the required TLS certificate.

  1. Generate your certificate authority (CA) RSA private key:

    openssl genrsa -out ca.key 4096
  2. Generate a self-signed CA certificate by using your CA key:

    openssl req -x509 -new -nodes -key ca.key -subj "/C=US/ST=North Carolina/L=Raleigh/O=Red Hat OpenShift" -days 400 -out ca.crt
  3. Generate the RSA private key for your certificate:

    openssl genrsa -out ingress.key 4096
  4. Generate the Certificate Signing request (CSR) by using the private key:

    openssl req -new -key ingress.key -out ingress.csr -config csr.cnf
  5. Generate a signed certificate by using your CA certificate and key and CSR:

    openssl x509 -req -in ingress.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out ingress.crt -sha256 -days 300 -extensions v3_ext -extfile csr.cnf
  6. Examine the certificate contents:

    openssl x509  -noout -text -in ./ingress.crt

1.2.2. Replace the Bring Your Own (BYO) ingress certificate

Complete the following steps to replace your BYO ingress certificate:

  1. Create the byo-ingress-tls secret by using your certificate and private key. Run the following command:

    oc -n open-cluster-management create secret tls byo-ingress-tls-secret --cert ./ingress.crt --key ./ingress.key
  2. Verify that the secret is created in the correct namespace with the following command:

    oc get secret -n open-cluster-management | grep -e byo-ingress-tls-secret -e byo-ca-cert
  3. Optional: Create a secret containing the CA certificate by running the following command:

    oc -n open-cluster-management create secret tls byo-ca-cert --cert ./ca.crt --key ./ca.key
  4. Delete the management-ingress subscription in order to redeploy the subscription. The secrets created in the previous steps are used automatically. Run the following command:

    oc delete subscription management-ingress-sub -n open-cluster-management
  5. Verify that the current certificate is your certificate, and that all console access and login functionality remain the same.

1.2.3. Restore the default self-signed certificate for management ingress

  1. Delete the bring your own certificate secrets with the following command:

    oc delete secret byo-ca-cert byo-ingress-tls-secret -n open-cluster-management
  2. Delete the management-ingress subscription in order to redeploy the subscription. The secrets created in the previous steps are used automatically. Run the following command:

    oc delete subscription management-ingress-sub -n open-cluster-management
  3. Verify that the current certificate is your certificate, and that all console access and login functionality remain the same.

See Certificates for more information about certificates that are created and managed by Red Hat Advanced Cluster Management. Return to the Risk and compliance page for more information on securing your cluster.