Chapter 6. Renewing the AMQ Interconnect certificate
Periodically, you must renew the CA certificate that secures the AMQ Interconnect connection between Red Hat OpenStack Platform (RHOSP) and Service Telemetry Framework (STF) when the certificate expires. The renewal is handled automatically by the cert-manager component in Red Hat OpenShift Container Platform, but you must manually copy the renewed certificate to your RHOSP nodes.
6.1. Checking for an expired AMQ Interconnect CA certificate
When the CA certificate expires, the AMQ Interconnect connections remain up, but cannot reconnect if they are interrupted. Eventually, some or all of the connections from your Red Hat OpenStack Platform (RHOSP) dispatch routers fail, showing errors on both sides, and the expiry or Not After field in your CA certificate is in the past.
Procedure
- Log in to Red Hat OpenShift Container Platform.
Change to the
service-telemetry
namespace:$ oc project service-telemetry
Verify that some or all dispatch router connections have failed:
$ oc exec -it $(oc get po -l application=default-interconnect -o jsonpath='{.items[0].metadata.name}') -- qdstat --connections | grep Router | wc 0 0 0
Check for this error in the Red Hat OpenShift Container Platform-hosted AMQ Interconnect logs:
$ oc logs -l application=default-interconnect | tail [...] 2022-11-10 20:51:22.863466 +0000 SERVER (info) [C261] Connection from 10.10.10.10:34570 (to 0.0.0.0:5671) failed: amqp:connection:framing-error SSL Failure: error:140940E5:SSL routines:ssl3_read_bytes:ssl handshake failure
- Log into your RHOSP undercloud.
Check for this error in the RHOSP-hosted AMQ Interconnect logs of a node with a failed connection:
$ ssh controller-0.ctlplane -- sudo tail /var/log/containers/metrics_qdr/metrics_qdr.log [...] 2022-11-10 20:50:44.311646 +0000 SERVER (info) [C137] Connection to default-interconnect-5671-service-telemetry.apps.mycluster.com:443 failed: amqp:connection:framing-error SSL Failure: error:0A000086:SSL routines::certificate verify failed
Confirm that the CA certificate has expired by examining the file on an RHOSP node:
$ ssh controller-0.ctlplane -- cat /var/lib/config-data/puppet-generated/metrics_qdr/etc/pki/tls/certs/CA_sslProfile.pem | openssl x509 -text | grep "Not After" Not After : Nov 10 20:31:16 2022 GMT $ date Mon Nov 14 11:10:40 EST 2022
6.2. Updating the AMQ Interconnect CA certificate
To update the AMQ Interconnect certificate, you must export it from Red Hat OpenShift Container Platform and copy it to your Red Hat OpenStack Platform (RHOSP) nodes.
Procedure
- Log in to Red Hat OpenShift Container Platform.
Change to the
service-telemetry
namespace:$ oc project service-telemetry
Export the CA certificate to
STFCA.pem
:$ oc get secret/default-interconnect-selfsigned -o jsonpath='{.data.ca\.crt}' | base64 -d > STFCA.pem
-
Copy
STFCA.pem
to your RHOSP undercloud. - Log into your RHOSP undercloud.
-
Edit the
stf-connectors.yaml
file to contain the new caCertFileContent. For more information, see Section 4.1.4, “Configuring the STF connection for the overcloud”. Copy the
STFCA.pem
file to each RHOSP overcloud node:[stack@undercloud-0 ~]$ ansible -i overcloud-deploy/overcloud/tripleo-ansible-inventory.yaml allovercloud -b -m copy -a "src=STFCA.pem dest=/var/lib/config-data/puppet-generated/metrics_qdr/etc/pki/tls/certs/CA_sslProfile.pem"
Restart the metrics_qdr container on each RHOSP overcloud node:
[stack@undercloud-0 ~]$ ansible -i overcloud-deploy/overcloud/tripleo-ansible-inventory.yaml allovercloud -m shell -a "sudo podman restart metrics_qdr"
NoteYou do not need to deploy the overcloud after you copy the
STFCA.pem
file and restart themetrics_qdr
container. You edit thestf-connectors.yaml
file so that future deployments do not overwrite the new CA certificate.