Chapter 4. Updating OpenShift Container Storage in external mode

Use the following procedures to update your OpenShift Container Storage cluster deployed in external mode.

Important

Upgrading Red Hat OpenShift Container Storage Operator does not upgrade the external Red Hat Ceph Storage cluster. It only upgrades the Red Hat OpenShift Container Storage Services running on the OpenShift Container Platform.

To upgrade the external Red Hat Ceph Storage cluster contact your Red Hat Ceph Storage administrator.

4.1. Enabling automatic updates for OpenShift Container Storage operator in external mode

Use this procedure to enable automatic update approval for updating OpenShift Container Storage operator in OpenShift Container Platform.

Note

Updating OpenShift Container Storage will not update the external Red Hat Ceph Storage cluster.

Prerequisites

  • Red Hat Ceph Storage version 4.2z1 or later is required for the external cluster. For more information, see this knowledge base article on Red Hat Ceph Storage releases and corresponding Ceph package versions.
  • Update the OpenShift Container Platform cluster to the latest stable release of version 4.5.X or 4.6.Y, see Updating Clusters.
  • Switch the Red Hat OpenShift Container Storage channel from stable-4.5 to stable-4.6. For details about channels, see OpenShift Container Storage upgrade channels and releases.

    Note

    You are required to switch channels only when you are updating minor versions (for example, updating from 4.5 to 4.6) and not when updating between batch updates of 4.6 (for example, updating from 4.6.0 to 4.6.1).

  • Ensure that all OpenShift Container Storage Pods, including the operator pods, are in Running state in the openshift-storage namespace.

    To view the state of the pods, click Workloads → Pods from the left pane of the OpenShift Web Console. Select openshift-storage from the Project drop down list.

  • Under Persistent Storage in the Status card, confirm that the OpenShift Container Storage cluster is healthy.
  • Ensure that you have sufficient time to complete the Openshift Container Storage update process.

Procedure

  1. Log in to OpenShift Web Console.
  2. Click OperatorsInstalled Operators
  3. Select the openshift-storage project.
  4. Click on the OpenShift Container Storage operator name.
  5. Click the Subscription tab and click the link under Approval.
  6. Select Automatic (default) and click Save.
  7. Perform one of the following depending on the Upgrade Status:

    • Upgrade Status shows requires approval.

      Note

      Upgrade status shows requires approval if the new OpenShift Container Storage version is already detected in the channel, and approval strategy was changed from Manual to Automatic at the time of update.

      1. Click the Install Plan link.
      2. On the InstallPlan Details page, click Preview Install Plan.
      3. Review the install plan and click Approve.
      4. Wait for the Status to change from Unknown to Created.
      5. Click OperatorsInstalled Operators
      6. Select the openshift-storage project.
      7. Wait for the Status to change to Up to date
    • Upgrade Status does not show requires approval:

      1. Wait for the update to initiate. This may take up to 20 minutes.
      2. Click OperatorsInstalled Operators
      3. Select the openshift-storage project.
      4. Wait for the Status to change to Up to date

Verification steps

  1. Click Overview → Persistent Storage tab and in Status card confirm that the OpenShift Container Storage cluster has a green tick mark indicating it is healthy.
  2. Click OperatorsInstalled OperatorsOpenShift Container Storage Operator. Under Storage Cluster, verify that the cluster service status in Ready.

    Note

    Once updated from OpenShift Container Storage version 4.5 to 4.6, the Version field here will still display 4.5. This is because the ocs-operator does not update the string represented in this field.

  3. Ensure that all OpenShift Container Storage Pods, including the operator pods, are in Running state in the openshift-storage namespace.

    To view the state of the pods, click Workloads → Pods from the left pane of the OpenShift Web Console. Select openshift-storage from the Project drop down list.

  4. If verification steps fail, contact Red Hat Support.

Next steps

  1. Enable monitoring to view the Object Gateway (RGW) metrics. See Enabling monitoring for the Object Service dashboard.

4.2. Manually updating OpenShift Container Storage operator in external mode

Use this procedure to update OpenShift Container Storage operator by providing manual approval to the install plan.

Note

Updating OpenShift Container Storage will not update the external Red Hat Ceph Storage cluster.

Prerequisites

  • Red Hat Ceph Storage version 4.2z1 or later is required for the external cluster. For more information, see this knowledge base article on Red Hat Ceph Storage releases and corresponding Ceph package versions.
  • Update the OpenShift Container Platform cluster to the latest stable release of version 4.5.X or 4.6.Y, see Updating Clusters.
  • Switch the Red Hat OpenShift Container Storage channel from stable-4.5 to stable-4.6. For details about channels, see OpenShift Container Storage upgrade channels and releases.

    Note

    You are required to switch channels only when you are updating minor versions (for example, updating from 4.5 to 4.6) and not when updating between batch updates of 4.6 (for example, updating from 4.6.0 to 4.6.1).

  • Ensure that all OpenShift Container Storage Pods, including the operator pods, are in Running state in the openshift-storage namespace.

    To view the state of the pods, click WorkloadsPods from the left pane of the OpenShift Web Console. Select openshift-storage from the Project drop down list.

  • Under Persistent Storage in Status card, confirm that the OpenShift Container Storage cluster is healthy.
  • Ensure that you have sufficient time to complete the Openshift Container Storage update process.

Procedure

  1. Log in to OpenShift Web Console.
  2. Click OperatorsInstalled Operators
  3. Select the openshift-storage project.
  4. Click the OpenShift Container Storage operator name.
  5. Click Subscription tab and click the link under Approval.
  6. Select Manual and click Save.
  7. Wait for the Upgrade Status to change to Upgrading.
  8. If the Upgrade Status shows requires approval, click on requires approval.
  9. On the InstallPlan Details page, click Preview Install Plan.
  10. Review the install plan and click Approve.
  11. Wait for the Status to change from Unknown to Created.
  12. Click OperatorsInstalled Operators
  13. Select the openshift-storage project.
  14. Wait for the Status to change to Up to date

Verification steps

  1. Click Overview → Persistent Storage tab and in Status card confirm that the OpenShift Container Storage cluster has a green tick mark indicating it is healthy.
  2. Click OperatorsInstalled OperatorsOpenShift Container Storage Operator. Under Storage Cluster, verify that the cluster service status in Ready.
  3. Ensure that all OpenShift Container Storage Pods, including the operator pods, are in Running state in the openshift-storage namespace.

    To view the state of the pods, click WorkloadsPods from the left pane of the OpenShift Web Console. Select openshift-storage from the Project drop down list.

    Note

    Once updated from OpenShift Container Storage version 4.5 to 4.6, the Version field here will still display 4.5. This is because the ocs-operator does not update the string represented in this field.

  4. If verification steps fail, contact Red Hat Support.

Next steps

  1. Enable monitoring to view the Object Gateway (RGW) metrics. See Enabling monitoring for the Object Service dashboard.

4.3. Enabling monitoring for the Object Service dashboard

In order to view the Object Gateway (RGW) metrics in the Object Service dashboard in OpenShift Container Storage 4.6, you must enable monitoring. For more information about the Object Service Dashboard, see Object Service dashboard indicators and Metrics in the Object Service dashboard.

Prerequisites

Procedure

  1. Download the python script for updating and extracting the Ceph cluster details.

    1. Click OperatorsInstalled Operators on the OpenShift Web Console to view the installed operators.
    2. On the Installed Operators page, click the Storage Cluster link in the Provided APIs column for the installed OpenShift Container Storage operator.
    3. Click Create Storage Cluster.
    4. Select Mode as External.
    5. In the Connect to external cluster section, click the Download Script link to download the python script for extracting the Ceph cluster details.
    6. Click Cancel as the storage cluster has already been created.
  2. Update the permission caps for the OpenShift Container Storage client on the Red Hat Ceph Storage cluster.

    1. Contact your Red Hat Ceph Storage administrator to run the downloaded python script on a Red Hat Ceph Storage client node.
    2. To run the script:

      # python3 ceph-external-cluster-details-exporter.py --upgrade --run-as-user=<client_name_used_for_OCS_4.5_install>

      --run-as-user provides the client name used during the OpenShift Container Storage 4.5 deployment. If this option was not used during the deployment of OpenShift Container Storage 4.5, the default client name client.healthchecker is set.

      The updated permissions for the user are set as:

      caps: [mgr] allow command config
      caps: [mon] allow r, allow command quorum_status, allow command version
      caps: [osd] allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow rx pool=default.rgw.log, allow x pool=default.rgw.buckets.index
  3. To enable monitoring for the Object Service Dashboard, the monitoring information must be added to the external secret rook-ceph-external-cluster-details in the openshift-storage namespace. There are two ways to update the secret:

    • To update the secret using the CLI, refer to this Red Hat Knowledgebase article.
    • To update the secret from the OpenShift Web Console, follow the steps below.

      1. Run the script downloaded in step 1v:

        # python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name <rbd block pool name> --monitoring-endpoint <ceph mgr prometheus exporter endpoint> --monitoring-endpoint-port <ceph mgr prometheus exporter port> --run-as-user <client_name_used_for_OCS_4.5_install>  [optional arguments]

        monitoring-endpoint is the IP address of the active ceph-mgr reachable from the OpenShift Container Platform cluster.

        monitoring-endpoint-port is the port associated with the ceph-mgr Prometheus exporter specified by --monitoring-endpoint. Only port 9283 is supported in OpenShift Container Storage 4.6.

        Note

        Ensure that all the parameters, including the optional arguments, except for monitoring-endpoint and monitoring-endpoint-port, are the same parameters that were used during the deployment of OpenShift Container Storage 4.5 in external mode.

      2. Save the JSON output generated after running the script in the previous step. Example output:

        [{"name": "rook-ceph-mon-endpoints", "kind": "ConfigMap", "data": {"data": "xxx.xxx.xxx.xxx:xxxx", "maxMonId": "0", "mapping": "{}"}}, {"name": "rook-ceph-mon", "kind": "Secret", "data": {"admin-secret": "admin-secret", "fsid": "<fs-id>", "mon-secret": "mon-secret"}}, {"name": "rook-ceph-operator-creds", "kind": "Secret", "data": {"userID": "client.healthchecker", "userKey": "<user-key>"}}, {"name": "rook-csi-rbd-node", "kind": "Secret", "data": {"userID": "csi-rbd-node", "userKey": "<user-key>"}}, {"name": "ceph-rbd", "kind": "StorageClass", "data": {"pool": "ceph-rbd"}}, {"name": "monitoring-endpoint", "kind": "CephCluster", "data": {"MonitoringEndpoint": "xxx.xxx.xxx.xxx", "MonitoringPort": "xxxx"}}, {"name": "rook-csi-rbd-provisioner", "kind": "Secret", "data": {"userID": "csi-rbd-provisioner", "userKey": "<user-key>"}}, {"name": "rook-csi-cephfs-provisioner", "kind": "Secret", "data": {"adminID": "csi-cephfs-provisioner", "adminKey": "<admin-key>"}}, {"name": "rook-csi-cephfs-node", "kind": "Secret", "data": {"adminID": "csi-cephfs-node", "adminKey": "<admin-key>"}}, {"name": "cephfs", "kind": "StorageClass", "data": {"fsName": "cephfs", "pool": "cephfs_data"}}, {"name": "ceph-rgw", "kind": "StorageClass", "data": {"endpoint": "xxx.xxx.xxx.xxx:xxxx", "poolPrefix": "default"}}]
      3. On the OpenShift Container Platform Web Console, under the openshift-storage namespace, navigate to WorkloadsSecrets and click rook-ceph-external-cluster-details:

        Example rook-ceph-external-cluster-details page
      4. From the Actions dropdown menu, click Edit Secret.
      5. On the Edit Key/Value Secret page, click Browse next to the Value text box and upload the JSON file saved previously.

        Example Edit Key/Value Secret page
      6. Click Save.

Verification Steps

  1. Click HomeOverview from the left pane of the OpenShift Web Console and click the Object Service tab.
  2. In the Status card, verify that the Object Service displays a green tick icon as shown in following image:

    Example of Object Service status verified
  3. In the Performance card, select Object Gateway (RGW) from the dropdown to view the metrics:

    Example Object Gateway (RGW) metrics
  4. Click OverviewPersistent Storage tab and in Status card confirm that the OpenShift Container Storage cluster has a green tick mark indicating it is healthy.