Chapter 7. Renewing the AMQ Interconnect certificate

Periodically, you must renew the CA certificate that secures the AMQ Interconnect connection between Red Hat OpenStack Platform (RHOSP) and Service Telemetry Framework (STF) when the certificate expires. The renewal is handled automatically by the cert-manager component in Red Hat OpenShift Container Platform, but you must manually copy the renewed certificate to your RHOSP nodes.

7.1. Checking for an expired AMQ Interconnect CA certificate

When the CA certificate expires, the AMQ Interconnect connections remain up, but cannot reconnect if they are interrupted. Eventually, some or all of the connections from your Red Hat OpenStack Platform (RHOSP) dispatch routers fail, showing errors on both sides, and the expiry or Not After field in your CA certificate is in the past.

Procedure

  1. Log in to Red Hat OpenShift Container Platform.

  2. Change to the service-telemetry namespace: 

    $ oc project service-telemetry

  3. Verify that some or all dispatch router connections have failed: 

    $ oc exec -it deploy/default-interconnect -- qdstat --connections | grep Router | wc
      0       0       0

  4. Check for this error in the Red Hat OpenShift Container Platform-hosted AMQ Interconnect logs: 

    $ oc logs -l application=default-interconnect | tail
[...]
2022-11-10 20:51:22.863466 +0000 SERVER (info) [C261] Connection from 10.10.10.10:34570 (to 0.0.0.0:5671) failed: amqp:connection:framing-error SSL Failure: error:140940E5:SSL routines:ssl3_read_bytes:ssl handshake failure
  5. Log into your RHOSP undercloud.

  6. Check for this error in the RHOSP-hosted AMQ Interconnect logs of a node with a failed connection: 

    $ ssh controller-0.ctlplane -- sudo tail /var/log/containers/metrics_qdr/metrics_qdr.log
[...]
2022-11-10 20:50:44.311646 +0000 SERVER (info) [C137] Connection to default-interconnect-5671-service-telemetry.apps.mycluster.com:443 failed: amqp:connection:framing-error SSL Failure: error:0A000086:SSL routines::certificate verify failed

  7. Confirm that the CA certificate has expired by examining the file on an RHOSP node: 

    $ ssh controller-0.ctlplane -- cat /var/lib/config-data/puppet-generated/metrics_qdr/etc/pki/tls/certs/CA_sslProfile.pem | openssl x509 -text | grep "Not After"
            Not After : Nov 10 20:31:16 2022 GMT

$ date
Mon Nov 14 11:10:40 EST 2022

7.2. Updating the AMQ Interconnect CA certificate

To update the AMQ Interconnect certificate, you must export it from Red Hat OpenShift Container Platform and copy it to your Red Hat OpenStack Platform (RHOSP) nodes.

Procedure

  1. Log in to Red Hat OpenShift Container Platform.

  2. Change to the service-telemetry namespace: 

    $ oc project service-telemetry

  3. Export the CA certificate to STFCA.pem: 

    $ oc get secret/default-interconnect-selfsigned -o jsonpath='{.data.ca\.crt}' | base64 -d > STFCA.pem
  4. Copy STFCA.pem to your RHOSP undercloud.
  5. Log into your RHOSP undercloud.
  6. Edit the stf-connectors.yaml file to contain the new caCertFileContent. For more information, see Section 4.1.5, “Configuring the STF connection for the overcloud”.

  7. Copy the STFCA.pem file to each RHOSP overcloud node: 

    [stack@undercloud-0 ~]$ ansible -i overcloud-deploy/overcloud/tripleo-ansible-inventory.yaml allovercloud -b -m copy -a "src=STFCA.pem dest=/var/lib/config-data/puppet-generated/metrics_qdr/etc/pki/tls/certs/CA_sslProfile.pem"

  8. Restart the metrics_qdr container on each RHOSP overcloud node: 

    [stack@undercloud-0 ~]$ ansible -i overcloud-deploy/overcloud/tripleo-ansible-inventory.yaml allovercloud -m shell -a "sudo podman restart metrics_qdr"
    Note

    You do not need to deploy the overcloud after you copy the STFCA.pem file and restart the metrics_qdr container. You edit the stf-connectors.yaml file so that future deployments do not overwrite the new CA certificate.