Chapter 8. Upgrading Service Telemetry Framework to version 1.5

To upgrade Service Telemetry Framework (STF) 1.4 to STF 1.5, you must complete the following steps:

  • Replace AMQ Certificate Manager with Certificate Manager.
  • Remove the ClusterServiceVersion and Subscription objects for Smart Gateway Operator and Service Telemetry Operator in the service-telemetry namespace on your Red Hat OpenShift Container Platform environment.
  • Upgrade Red Hat OpenShift Container Platform from 4.8 to 4.10.
  • Re-enable the operators that you removed.
  • Update the AMQ Interconnect CA Certificate on Red Hat OpenStack Platform (RHOSP).

Prerequisites

  • You have backed up your data. There is an outage during the Red Hat OpenShift Container Platform upgrade. You cannot reconfigure the ServiceTelemetry and SmartGateway objects during the Operators replacement.
  • You have prepared your environment for upgrade from Red Hat OpenShift Container Platform 4.8 to the supported version, 4.10.
  • The Red Hat OpenShift Container Platform cluster is fully-connected. STF does not support disconnected or restricted-network clusters.

8.1. Removing the Service Telemetry Framework 1.4 Operators

Remove the Service Telemetry Framework (STF) 1.4 Operators and the AMQ Certificate Manager Operator from the Red Hat OpenShift Container Platform 4.8.

Procedure

  1. Remove the Service Telemetry Operator.
  2. Remove the Smart Gateway Operator.
  3. Remove the AMQ Certificate Manager Operator.
  4. Remove the Grafana Operator.

Additional resources

8.1.1. Removing the Service Telemetry Operator

As part of upgrading your Service Telemetry Framework (STF) installation, you must remove the Service Telemetry Operator in the service-telemetry namespace on your Red Hat OpenShift Container Platform environment.

Procedure

  1. Change to the service-telemetry project: 

    $ oc project service-telemetry

  2. Remove the Service Telemetry Operator Subscription: 

    $ oc delete sub --selector=operators.coreos.com/service-telemetry-operator.service-telemetry

subscription.operators.coreos.com "service-telemetry-operator" deleted

  3. Remove the Service Telemetry Operator ClusterServiceVersion: 

    $ oc delete csv --selector=operators.coreos.com/service-telemetry-operator.service-telemetry

clusterserviceversion.operators.coreos.com "service-telemetry-operator.v1.4.1669718959" deleted

Verification

  1. Verify that the Service Telemetry Operator deployment is not running: 

    $ oc get deploy --selector=operators.coreos.com/service-telemetry-operator.service-telemetry

No resources found in service-telemetry namespace.

  2. Verify the Service Telemetry Operator subscription is absent: 

    $ oc get sub --selector=operators.coreos.com/service-telemetry-operator.service-telemetry

No resources found in service-telemetry namespace.

  3. Verify the Service Telemetry Operator ClusterServiceVersion is absent: 

    $ oc get csv --selector=operators.coreos.com/service-telemetry-operator.service-telemetry

No resources found in service-telemetry namespace.

8.1.2. Removing the Smart Gateway Operator

As part of upgrading your Service Telemetry Framework (STF) installation, you must remove the Smart Gateway Operator in the service-telemetry namespace on your Red Hat OpenShift Container Platform environment.

Procedure

  1. Change to the service-telemetry project: 

    $ oc project service-telemetry

  2. Remove the Smart Gateway Operator Subscription: 

    $ oc delete sub --selector=operators.coreos.com/smart-gateway-operator.service-telemetry

subscription.operators.coreos.com "smart-gateway-operator-stable-1.4-redhat-operators-openshift-marketplace" deleted

  3. Remove the Smart Gateway Operator ClusterServiceVersion: 

    $ oc delete csv --selector=operators.coreos.com/smart-gateway-operator.service-telemetry

clusterserviceversion.operators.coreos.com "smart-gateway-operator.v4.0.1669718962" deleted

Verification

  1. Verify that the Smart Gateway Operator deployment is not running: 

    $ oc get deploy --selector=operators.coreos.com/smart-gateway-operator.service-telemetry

No resources found in service-telemetry namespace.

  2. Verify the Smart Gateway Operator subscription is absent: 

    $ oc get sub --selector=operators.coreos.com/smart-gateway-operator.service-telemetry

No resources found in service-telemetry namespace.

  3. Verify the Smart Gateway Operator ClusterServiceVersion is absent: 

    $ oc get csv --selector=operators.coreos.com/smart-gateway-operator.service-telemetry

No resources found in service-telemetry namespace.

8.1.3. Removing the AMQ Certificate Manager Operator

Procedure

  1. Remove the AMQ Certificate Manager Operator Subscription: 

    $ oc delete sub --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators

subscription.operators.coreos.com "amq7-cert-manager-operator" deleted

  2. Remove the AMQ Certificate Manager Operator ClusterServiceVersion: 

    $ oc delete csv --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators

clusterserviceversion.operators.coreos.com "amq7-cert-manager.v1.0.11" deleted

Verification

  1. Verify that the AMQ Certificate Manager Operator deployment is not running: 

    $ oc get deploy --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators

No resources found in openshift-operators namespace.

  2. Verify that the AMQ Certificate Manager Operator subscription is absent: 

    $ oc get sub --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.service-telemetry

No resources found in openshift-operators namespace.

  3. Verify that the AMQ Certificate Manager Operator Cluster Service Version is absent: 

    $ oc get csv --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators

No resources found in openshift-operators namespace.

8.1.4. Removing the Grafana Operator

Procedure

  1. Remove the Grafana Operator Subscription: 

    $ oc delete sub --selector=operators.coreos.com/grafana-operator.service-telemetry

subscription.operators.coreos.com "grafana-operator" deleted

  2. Remove the Grafana Operator ClusterServiceVersion: 

    $ oc delete csv --selector=operators.coreos.com/grafana-operator.service-telemetry

clusterserviceversion.operators.coreos.com "grafana-operator.v3.10.3" deleted

Verification

  1. Verify the Grafana Operator deployment is not running: 

    $ oc get deploy --selector=operators.coreos.com/grafana-operator.service-telemetry

No resources found in service-telemetry namespace.

  2. Verify the Grafana Operator subscription is absent: 

    $ oc get sub --selector=operators.coreos.com/grafana-operator.service-telemetry

No resources found in service-telemetry namespace.

  3. Verify the Grafana Operator Cluster Service Version is absent: 

    $ oc get csv --selector=operators.coreos.com/grafana-operator.service-telemetry

No resources found in service-telemetry namespace.

8.2. Upgrading Red Hat OpenShift Container Platform to 4.10

Service Telemetry Framework (STF) 1.5 is only compatible with Red Hat OpenShift Container Platform 4.10. For more information about upgrading your Red Hat OpenShift Container Platform from 4.8 to 4.10, see Updating clusters overview.

Prerequisites

  • You removed the STF 1.4 Operators.
  • You removed the AMQ Certificate Manager Operator and Grafana Operator. You must remove the Operators before you upgrade Red Hat OpenShift Container Platform because the Operator APIs are incompatible with 4.10. For more information about preparing your Red Hat OpenShift Container Platform for upgrade from 4.8 to 4.10, see Understanding OpenShift Container Platform updates.

  • Verify the suitability of your Red Hat OpenShift Container Platform upgrade: 

    $ oc adm upgrade

    You cannot upgrade the cluster if you encounter the following error message: 

    Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: service-telemetry/grafana-operator.v3.10.3 is incompatible with OpenShift minor versions greater than 4.8,openshift-operators/amq7-cert-manager.v1.0.11 is incompatible with OpenShift minor versions greater than 4.8

8.3. Installing the Service Telemetry Framework 1.5 Operators

Install the Service Telemetry Framework (STF) 1.5 Operators and the Certificate Manager for OpenShift Operator on your Red Hat OpenShift Container Platform environment. See Section 1.1, “Support for Service Telemetry Framework” for more information about STF support status and life cycle.

Note

After a successful STF 1.5 install, you must retrieve and apply the AMQ Interconnect CA certificate to the Red Hat OpenStack Platform environment, or the transport layer and telemetry data becomes unavailable.

For more information about updating the AMQ Interconnect CA certificate, see Section 8.4, “Updating the AMQ Interconnect CA Certificate on Red Hat OpenStack Platform”.

Prerequisites

Procedure

  1. Change to the service-telemetry project: 

    $ oc project service-telemetry

  2. Create a namespace for the cert-manager Operator: 

    $ oc create -f - <<EOF
apiVersion: project.openshift.io/v1
kind: Project
metadata:
  name: openshift-cert-manager-operator
spec:
  finalizers:
  - kubernetes
EOF

  3. Create an OperatorGroup for the cert-manager Operator: 

    $ oc create -f - <<EOF
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: openshift-cert-manager-operator
  namespace: openshift-cert-manager-operator
spec: {}
EOF

  4. Subscribe to the cert-manager Operator with the redhat-operators CatalogSource: 

    $ oc create -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: openshift-cert-manager-operator
  namespace: openshift-cert-manager-operator
spec:
  channel: tech-preview
  installPlanApproval: Automatic
  name: openshift-cert-manager-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

  5. Validate your ClusterServiceVersion. Ensure that the phase of cert-manager Operator is Succeeded: 

    $ oc get csv --namespace openshift-cert-manager-operator --selector=operators.coreos.com/openshift-cert-manager-operator.openshift-cert-manager-operator

NAME                            DISPLAY                                       VERSION   REPLACES   PHASE
openshift-cert-manager.v1.7.1   cert-manager Operator for Red Hat OpenShift   1.7.1-1              Succeeded
  6. Optional: Resubscribe to the Grafana Operator. For more information, see: test Section 5.1.1, “Configuring Grafana to host the dashboard”.

  7. Create the Service Telemetry Operator subscription to manage the STF instances: 

    $ oc create -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: service-telemetry-operator
  namespace: service-telemetry
spec:
  channel: stable-1.5
  installPlanApproval: Automatic
  name: service-telemetry-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

  8. Validate the Service Telemetry Operator and the dependent operators: 

    $ oc get csv --namespace service-telemetry

NAME                                          DISPLAY                                       VERSION          REPLACES                                      PHASE
amq7-interconnect-operator.v1.10.13           Red Hat Integration - AMQ Interconnect        1.10.13          amq7-interconnect-operator.v1.10.4            Succeeded
elasticsearch-eck-operator-certified.v2.5.0   Elasticsearch (ECK) Operator                  2.5.0            elasticsearch-eck-operator-certified.v2.4.0   Succeeded
openshift-cert-manager.v1.7.1                 cert-manager Operator for Red Hat OpenShift   1.7.1-1                                                        Succeeded
prometheusoperator.0.47.0                     Prometheus Operator                           0.47.0           prometheusoperator.0.37.0                     Succeeded
service-telemetry-operator.v1.5.1669950702    Service Telemetry Operator                    1.5.1669950702                                                 Succeeded
smart-gateway-operator.v5.0.1669950681        Smart Gateway Operator                        5.0.1669950681                                                 Succeeded

Verification

  • Verify that the Service Telemetry Operator has successfully reconciled. 

    $ oc logs -f --selector=name=service-telemetry-operator

[...]
----- Ansible Task Status Event StdOut (infra.watch/v1beta1, Kind=ServiceTelemetry, default/service-telemetry) -----


PLAY RECAP *********************************************************************
localhost                  : ok=115  changed=0    unreachable=0    failed=0    skipped=21   rescued=0    ignored=0


$ oc get pods
NAME                                                      READY   STATUS    RESTARTS        AGE
alertmanager-default-0                                    3/3     Running   0               20h
default-cloud1-ceil-event-smartgateway-6d57ffbbdc-5mrj8   2/2     Running   1 (3m42s ago)   4m21s
default-cloud1-ceil-meter-smartgateway-67684d88c8-62mp7   3/3     Running   1 (3m43s ago)   4m20s
default-cloud1-coll-event-smartgateway-66cddd5567-qb6p2   2/2     Running   1 (3m42s ago)   4m19s
default-cloud1-coll-meter-smartgateway-76d5ff6db5-z5ch8   3/3     Running   0               4m20s
default-cloud1-sens-meter-smartgateway-7b59669fdd-c42zg   3/3     Running   1 (3m43s ago)   4m20s
default-interconnect-845c4b647c-wwfcm                     1/1     Running   0               4m10s
elastic-operator-57b57964c5-6q88r                         1/1     Running   8 (17h ago)     20h
elasticsearch-es-default-0                                1/1     Running   0               21h
grafana-deployment-59c54f7d7c-zjnhm                       2/2     Running   0               20h
interconnect-operator-848889bf8b-bq2zx                    1/1     Running   0               20h
prometheus-default-0                                      3/3     Running   1 (20h ago)     20h
prometheus-operator-5d7b69fd46-c2xlv                      1/1     Running   0               20h
service-telemetry-operator-79fb664dfb-nj8jn               1/1     Running   0               5m11s
smart-gateway-operator-79557664f8-ql7xr                   1/1     Running   0               5m7s

8.4. Updating the AMQ Interconnect CA Certificate on Red Hat OpenStack Platform

After you upgrade to Service Telemetry Framework (STF) v1.5, the CA certificate for AMQ Interconnect regenerates. In STF v1.4 the CA certificate for AMQ Interconnect is valid for three months and must be updated periodically in Red Hat OpenStack Platform (RHOSP). In STF v1.5, the generated certificates are valid for the lifetime of the RHOSP lifecycle, 70080 hours by default.

Prerequisites

  • You have successfully installed STF v1.5 and updated the CA certificate for AMQ Interconnect.

Procedure