Upgrading to OpenShift Data Foundation

Red Hat OpenShift Data Foundation 4.9

Instructions on how to update OpenShift Container Storage to OpenShift Data Foundation latest version.

Red Hat Storage Documentation Team

Abstract

This document explains how to update previous versions of OpenShift Container Storage to Red Hat OpenShift Data Foundation.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.

Providing feedback on Red Hat documentation

We appreciate your input on our documentation. Do let us know how we can make it better. To give feedback:

  • For simple comments on specific passages:

    1. Make sure you are viewing the documentation in the Multi-page HTML format. In addition, ensure you see the Feedback button in the upper right corner of the document.
    2. Use your mouse cursor to highlight the part of text that you want to comment on.
    3. Click the Add Feedback pop-up that appears below the highlighted text.
    4. Follow the displayed instructions.
  • For submitting more complex feedback, create a Bugzilla ticket:

    1. Go to the Bugzilla website.
    2. In the Component section, choose documentation.
    3. Fill in the Description field with your suggestion for improvement. Include a link to the relevant part(s) of documentation.
    4. Click Submit Bug.

Chapter 1. Overview of the OpenShift Data Foundation update process

OpenShift Container Storage, based on the open source Ceph technology, has expanded its scope and foundational role in a containerized, hybrid cloud environment since its introduction. It complements existing storage in addition to other data-related hardware and software, making them rapidly attachable, accessible, and scalable in a hybrid cloud environment. To better reflect these foundational and infrastructure distinctives, OpenShift Container Storage is now OpenShift Data Foundation.

Important

You can perform the upgrade process for OpenShift Data Foundation version 4.9 from OpenShift Container Storage version 4.8 only by installing the OpenShift Data Foundation operator from OpenShift Container Platform OperatorHub.

In the future release, you can upgrade Red Hat OpenShift Data Foundation, either between minor releases like 4.9 and 4.x, or between batch updates like 4.9.0 and 4.9.1 by enabling automatic updates (if not done so during operator installation) or performing manual updates.

You also need to upgrade the different parts of Red Hat OpenShift Data Foundation in the following order for both internal and external mode deployments:

  1. Update OpenShift Container Platform according to the Updating clusters documentation for OpenShift Container Platform.
  2. Update Red Hat OpenShift Data Foundation.

    1. To prepare a disconnected environment for updates, see Operators guide to using Operator Lifecycle Manager on restricted networks to be able to update Red Hat OpenShift Data Foundation as well as Local Storage Operator when in use.
    2. Update Red Hat OpenShift Container Storage operator version 4.8 to version 4.9 by installing the Red Hat OpenShift Data Foundation operator from the OperatorHub on OpenShift Container Platform web console. See Updating Red Hat OpenShift Container Storage 4.8 to Red Hat OpenShift Data Foundation 4.9.
    3. Update Red Hat OpenShift Data Foundation from 4.9.x to 4.9.y. See Updating Red Hat OpenShift Data Foundation 4.9.x to 4.9.y.
    4. For updating external mode deployments, you must also perform the steps from section Updating the OpenShift Data Foundation external secret.
    5. If you use local storage:

      1. Update the Local Storage operator.

        See Checking for Local Storage Operator deployments if you are unsure.

      2. Perform post-update configuration changes for clusters backed by local storage.

        See Post-update configuration for clusters backed by local storage for details.

Update considerations

Review the following important considerations before you begin.

  • Red Hat recommends using the same version of Red Hat OpenShift Container Platform with Red Hat OpenShift Data Foundation.

    See the Interoperability Matrix for more information about supported combinations of OpenShift Container Platform and Red Hat OpenShift Data Foundation.

  • To know whether your cluster was deployed in internal or external mode, refer to the knowledgebase article on How to determine if ODF cluster has storage in internal or external mode.
  • The Local Storage Operator is fully supported only when the Local Storage Operator version matches the Red Hat OpenShift Container Platform version.
  • The flexible scaling feature is available only in new deployments of Red Hat OpenShift Data Foundation versions 4.7 and later. Storage clusters upgraded from a previous version to version 4.7 or later do not support flexible scaling. For more information, see Flexible scaling of OpenShift Container Storage cluster in the New features section of 4.7 Release Notes.

Chapter 2. OpenShift Data Foundation upgrade channels and releases

In OpenShift Container Platform 4.1, Red Hat introduced the concept of channels for recommending the appropriate release versions for cluster upgrades. By controlling the pace of upgrades, these upgrade channels allow you to choose an upgrade strategy. As OpenShift Data Foundation gets deployed as an operator in OpenShift Container Platform, it follows the same strategy to control the pace of upgrades by shipping the fixes in multiple channels. Upgrade channels are tied to a minor version of OpenShift Data Foundation.

For example, OpenShift Data Foundation 4.9 upgrade channels recommend upgrades within 4.9. They do not recommend upgrades to 4.10 or later releases. This strategy ensures that administrators can explicitly decide to upgrade to the next minor version of OpenShift Data Foundation.

Upgrade channels control only release selection and do not impact the version of the cluster that you install; the odf-operator decides the version of OpenShift Data Foundation to be installed. Out of the box, it always installs the latest OpenShift Data Foundation release maintaining the compatibility with OpenShift Container Platform. So on OpenShift Container Platform 4.9, OpenShift Data Foundation 4.9 will be the latest version which can be installed.

OpenShift Data Foundation upgrades are tied to the OpenShift Container Platform upgrade to ensure that compatibility and interoperability are maintained with the OpenShift Container Platform. For OpenShift Data Foundation 4.9, OpenShift Container Platform 4.9 and 4.10 (when generally available) are supported. OpenShift Container Platform 4.10 is supported to maintain forward compatibility with OpenShift Container Platform. Keep the OpenShift Data Foundation version the same as OpenShift Container Platform in order to get the benefit of all the features and enhancements in that release.

Important

Due to fundamental Kubernetes design, all OpenShift Container Platform updates between minor versions must be serialized. You must update from OpenShift Container Platform 4.8 to 4.9 and then to 4.10. You cannot update from OpenShift Container Platform 4.8 to 4.10 directly. For more information, see Preparing to perform an EUS-to-EUS update of the Updating clusters guide in OpenShift Container Platform documentation.

OpenShift Data Foundation 4.9 offers the following upgrade channel:

  • stable-4.9
  • eus-4.y (only when running an even-numbered 4.y cluster release, like 4.10)

stable-4.9 channel

Once a new version is Generally Available, the stable channel corresponding to the minor version gets updated with the new image which can be used to upgrade. You can use the stable-4.9 channel to upgrade from OpenShift Container Storage 4.8 and upgrades within 4.9.

eus-4.y channel

In addition to the stable channel, all even-numbered minor versions of OpenShift Data Foundation offer Extended Update Support (EUS). These EUS versions extend the Full and Maintenance support phases for customers with Standard and Premium Subscriptions to 18 months. The only difference between stable-4.y and eus-4.y channels is that the EUS channels will include the release only when the next EUS release is available.

Chapter 3. Updating Red Hat OpenShift Container Storage 4.8 to Red Hat OpenShift Data Foundation 4.9

This chapter helps you to upgrade between the z-stream release for all Red Hat OpenShift Data Foundation deployments (Internal, Internal-Attached and External). The upgrade process remains the same for all deployments. The Only difference is what gets upgraded and what’s not.

  • For Internal and Internal-attached deployments, upgrading OpenShift Container Storage upgrades all OpenShift Container Storage services including the backend Ceph Storage cluster.
  • For External mode deployments, upgrading OpenShift Container Storage only upgrades the OpenShift Container Storage service while the backend Ceph storage cluster remains untouched and needs to be upgraded separately.

    We recommend upgrading RHCS along with OpenShift Container Storage in order to get new feature support, security fixes, and other bug fixes. Since we do not have a strong dependency on RHCS upgrade, you can upgrade the OpenShift Data Foundation operator first followed by RHCS upgrade or vice-versa. See solution to know more about Red Hat Ceph Storage releases.

Important

Upgrading to 4.9 directly from any version older than 4.8 is unsupported.

Prerequisites

  • Ensure that the OpenShift Container Platform cluster has been updated to the latest stable release of version 4.9.X, see Updating Clusters.
  • Ensure that the OpenShift Container Storage cluster is healthy and data is resilient.

    • Navigate to Storage → Overview and check both Block and File and Object tabs for the green tick on the status card. Green tick indicates that the storage cluster, object service and data resiliency are all healthy.
  • Ensure that all OpenShift Container Storage Pods, including the operator pods, are in Running state in the openshift-storage namespace.

    To view the state of the pods, on the OpenShift Web Console, click Workloads → Pods. Select openshift-storage from the Project drop-down list.

    Note

    If the Show default projects option is disabled, use the toggle button to list all the default projects.

  • Ensure that you have sufficient time to complete the OpenShift Data Foundation update process, as the update time varies depending on the number of OSDs that run in the cluster.

Procedure

  1. On the OpenShift Web Console, navigate to OperatorHub.
  2. Search for OpenShift Data Foundation using the Filter by keyword box and click on the OpenShift Data Foundation tile.
  3. Click Install.
  4. On the install Operator page, click Install. Wait for the Operator installation to complete.

    Note

    We recommend using all default settings. Changing it may result in unexpected behavior. Alter only if you are aware of its result.

Verification steps

  1. Verify that the page displays Succeeded message along with the option to Create StorageSystem.

    Note

    For the upgraded clusters, since the storage system is automatically created, do not create it again.

  2. On the notification popup, click Refresh web console link to reflect the OpenShift Data Foundation changes in the OpenShift console.
  3. Verify the state of the pods on the OpenShift Web Console.

    • Click Workloads → Pods.
    • Select openshift-storage from the Project drop-down list.

      Note

      If the Show default projects option is disabled, use the toggle button to list all the default projects.

      Wait for all the pods in the openshift-storage namespace to restart and reach Running state.

  4. Verify that the OpenShift Data Foundation cluster is healthy and data is resilient.

    • Navigate to StorageOpenShift Data foundationStorage Systems tab and then click on the storage system name.
    • Check both Block and File and Object tabs for the green tick on the status card. Green tick indicates that the storage cluster, object service and data resiliency are all healthy.
Important

Additional Resources

If you face any issues while updating OpenShift Data Foundation, see the Commonly required logs for troubleshooting section in the Troubleshooting guide.

Chapter 4. Updating Red Hat OpenShift Data Foundation 4.9.x to 4.9.y

This chapter helps you to upgrade between the z-stream release for all Red Hat OpenShift Data Foundation deployments (Internal, Internal-Attached and External). The upgrade process remains the same for all deployments. The Only difference is what gets upgraded and what’s not.

  • For Internal and Internal-attached deployments, upgrading OpenShift Container Storage upgrades all OpenShift Container Storage services including the backend Ceph Storage cluster.
  • For External mode deployments, upgrading OpenShift Container Storage only upgrades the OpenShift Container Storage service while the backend Ceph storage cluster remains untouched and needs to be upgraded separately.

    Hence, we recommend upgrading RHCS along with OpenShift Container Storage in order to get new feature support, security fixes, and other bug fixes. Since we do not have a strong dependency on RHCS upgrade, you can upgrade the OpenShift Data Foundation operator first followed by RHCS upgrade or vice-versa. See solution to know more about Red Hat Ceph Storage releases.

When a new z-stream release becomes available, the upgrade process triggers automatically if the update strategy was set to Automatic. If the update strategy is set to Manual then use the following procedure.

Prerequisites

  • Ensure that the OpenShift Container Platform cluster has been updated to the latest stable release of version 4.9.X, see Updating Clusters.
  • Ensure that the OpenShift Data Foundation cluster is healthy and data is resilient.

    • Navigate to Storage → OpenShift Data Foundation → Storage Systems tab and then click on the storage system name.
    • Check for the green tick on the status card of Overview - Block and File and Object tabs. Green tick indicates that the storage cluster, object service and data resiliency is healthy.
  • Ensure that all OpenShift Data Foundation Pods, including the operator pods, are in Running state in the openshift-storage namespace.

    To view the state of the pods, on the OpenShift Web Console, click Workloads → Pods. Select openshift-storage from the Project drop-down list.

    Note

    If the Show default projects option is disabled, use the toggle button to list all the default projects.

  • Ensure that you have sufficient time to complete the OpenShift Data Foundation update process, as the update time varies depending on the number of OSDs that run in the cluster.

Procedure

  1. On the OpenShift Web Console, navigate to Operators → Installed Operators.
  2. Select openshift-storage project.

    Note

    If the Show default projects option is disabled, use the toggle button to list all the default projects.

  3. Click the OpenShift Data Foundation operator name.
  4. Click the Subscription tab.
  5. If the Upgrade Status shows require approval, click on requires approval link.
  6. On the InstallPlan Details page, click Preview Install Plan.
  7. Review the install plan and click Approve.
  8. Wait for the Status to change from Unknown to Created.

Verification steps

  • Verify that the Version below the OpenShift Data Foundation name and the operator status is the latest version.

    • Navigate to Operators → Installed Operators and select the openshift-storage project.
    • When the upgrade completes, the version updates to a new version number for OpenShift Data Foundation and status changes to Succeeded with a green tick.
  • Verify that the OpenShift Data Foundation cluster is healthy and data is resilient.

    • Navigate to Storage → OpenShift Data Foundation → Storage Systems tab and then click on the storage system name.
    • Check for the green tick on the status card of Overview - Block and File and Object tabs. Green tick indicates that the storage cluster, object service and data resiliency is healthy
Important

In case the console plugin option was not automatically enabled after you installed the OpenShift Data Foundation Operator, you need to enable it.

For more information on how to enable the console plugin, see Enabling the Red Hat OpenShift Data Foundation console plugin.

Chapter 5. Changing the update approval strategy

To ensure that the storage system gets updated automatically when a new update is available in the same channel, we recommend keeping the update approval strategy to Automatic. Changing the update approval strategy to Manual will need manual approval for each upgrade.

Procedure

  1. Navigate to Operators → Installed Operators.
  2. Select openshift-storage from the Project drop-down list.

    Note

    If the Show default projects option is disabled, use the toggle button to list all the default projects.

  3. Click on OpenShift Data Foundation operator name
  4. Go to the Subscription tab.
  5. Click on the pencil icon for changing the Update approval.
  6. Select the update approval strategy and click Save.

Verification steps

  • Verify that the Update approval shows the newly selected approval strategy below it.

Chapter 6. Updating the OpenShift Data Foundation external secret

Update the OpenShift Data Foundation external secret after updating to the latest version of OpenShift Data Foundation.

Note

Updating the external secret is not required for batch updates. For example, when updating from OpenShift Data Foundation 4.9.X to 4.9.Y.

Prerequisites

  • Update the OpenShift Container Platform cluster to the latest stable release of 4.9.z, see Updating Clusters.
  • The OpenShift Container Storage operator has been upgraded to OpenShift Data Foundation version 4.9. See Updating Red Hat OpenShift Container Storage 4.8 to Red Hat OpenShift Data Foundation for more information.
  • Ensure that the OpenShift Data Foundation cluster is healthy and the data is resilient. Navigate to StorageOpenShift Data foundationStorage Systems tab and then click on the storage system name.

    • On the Overview - Block and File tab, check the Status card and confirm that the Storage cluster has a green tick indicating it is healthy.
    • Click the Object tab and confirm Object Service and Data resiliency has a green tick indicating it is healthy. The RADOS Object Gateway is only listed in case RADOS Object Gateway endpoint details are included while deploying OpenShift Data Foundation in external mode.
  • Red Hat Ceph Storage must have a Ceph dashboard installed and configured.

Procedure

  1. Download the OpenShift Data Foundation version of the ceph-external-cluster-details-exporter.py python script.

    # oc get csv $(oc get csv -n openshift-storage | grep ocs-operator | awk '{print $1}') -n openshift-storage -o jsonpath='{.metadata.annotations.external\.features\.ocs\.openshift\.io/export-script}' | base64 --decode > ceph-external-cluster-details-exporter.py
  2. Update permission caps on the external Red Hat Ceph Storage cluster by running ceph-external-cluster-details-exporter.py on any client node in the external Red Hat Ceph Storage cluster. You may need to ask your Red Hat Ceph Storage administrator to do this.

    # python3 ceph-external-cluster-details-exporter.py --upgrade --run-as-user=<ocs_client_name>
    --run-as-user

    The client name used during OpenShift Data Foundation cluster deployment. Use the default client name client.healthchecker if a different client name was not set.

    The updated permissions for the user are set as:

    caps: [mgr] allow command config
    caps: [mon] allow r, allow command quorum_status, allow command version
    caps: [osd] allow rwx pool=RGW_POOL_PREFIX.rgw.meta, allow r pool=.rgw.root, allow rw pool=RGW_POOL_PREFIX.rgw.control, allow rx pool=RGW_POOL_PREFIX.rgw.log, allow x pool=RGW_POOL_PREFIX.rgw.buckets.index
  3. Run the previously downloaded python script and save the JSON output that gets generated, from the external Red Hat Ceph Storage cluster.

    1. Run the previously downloaded python script:

      # python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name <rbd block pool name> --monitoring-endpoint <ceph mgr prometheus exporter endpoint> --monitoring-endpoint-port <ceph mgr prometheus exporter port> --rgw-endpoint <rgw endpoint> --run-as-user <ocs_client_name>  [optional arguments]
      --rbd-data-pool-name
      Is a mandatory parameter used for providing block storage in OpenShift Data Foundation.
      --rgw-endpoint
      Is optional. Provide this parameter if object storage is to be provisioned through Ceph Rados Gateway for OpenShift Data Foundation. Provide the endpoint in the following format: <ip_address>:<port>.
      --monitoring-endpoint
      Is optional. It accepts comma separated list of IP addresses of active and standby mgrs reachable from the OpenShift Container Platform cluster. If not provided, the value is automatically populated.
      --monitoring-endpoint-port
      Is optional. It is the port associated with the ceph-mgr Prometheus exporter specified by --monitoring-endpoint. If not provided, the value is automatically populated.
      --run-as-user

      The client name used during OpenShift Data Foundation cluster deployment. Use the default client name client.healthchecker if a different client name was not set.

      Note

      Ensure that all the parameters, including the optional arguments, except for monitoring-endpoint and monitoring-endpoint-port, are the same as what was used during the deployment of OpenShift Data Foundation in external mode.

    2. Save the JSON output generated after running the script in the previous step.

      Example output:

      [{"name": "rook-ceph-mon-endpoints", "kind": "ConfigMap", "data": {"data": "xxx.xxx.xxx.xxx:xxxx", "maxMonId": "0", "mapping": "{}"}}, {"name": "rook-ceph-mon", "kind": "Secret", "data": {"admin-secret": "admin-secret", "fsid": "<fs-id>", "mon-secret": "mon-secret"}}, {"name": "rook-ceph-operator-creds", "kind": "Secret", "data": {"userID": "<user-id>", "userKey": "<user-key>"}}, {"name": "rook-csi-rbd-node", "kind": "Secret", "data": {"userID": "csi-rbd-node", "userKey": "<user-key>"}}, {"name": "ceph-rbd", "kind": "StorageClass", "data": {"pool": "<pool>"}}, {"name": "monitoring-endpoint", "kind": "CephCluster", "data": {"MonitoringEndpoint": "xxx.xxx.xxx.xxxx", "MonitoringPort": "xxxx"}}, {"name": "rook-ceph-dashboard-link", "kind": "Secret", "data": {"userID": "ceph-dashboard-link", "userKey": "<user-key>"}}, {"name": "rook-csi-rbd-provisioner", "kind": "Secret", "data": {"userID": "csi-rbd-provisioner", "userKey": "<user-key>"}}, {"name": "rook-csi-cephfs-provisioner", "kind": "Secret", "data": {"adminID": "csi-cephfs-provisioner", "adminKey": "<admin-key>"}}, {"name": "rook-csi-cephfs-node", "kind": "Secret", "data": {"adminID": "csi-cephfs-node", "adminKey": "<admin-key>"}}, {"name": "cephfs", "kind": "StorageClass", "data": {"fsName": "cephfs", "pool": "cephfs_data"}}, {"name": "ceph-rgw", "kind": "StorageClass", "data": {"endpoint": "xxx.xxx.xxx.xxxx", "poolPrefix": "default"}}, {"name": "rgw-admin-ops-user", "kind": "Secret", "data": {"accessKey": "<access-key>", "secretKey": "<secret-key>"}}]
  4. Upload the generated JSON file.

    1. Log in to the OpenShift Web Console.
    2. Click Workloads → Secrets.
    3. Set project to openshift-storage.
    4. Click rook-ceph-external-cluster-details.
    5. Click Actions (⋮) → Edit Secret.
    6. Click Browse and upload the JSON file.
    7. Click Save.

Verification steps

  • To verify that the OpenShift Data Foundation cluster is healthy and data is resilient, navigate to StorageOpenShift Data foundationStorage Systems tab and then click on the storage system name.

    • On the OverviewBlock and File tab, check the Details card to verify that the RHCS dashboard link is available and also check the Status card to confirm that the Storage Cluster has a green tick indicating it is healthy.
    • Click the Object tab and confirm Object Service and Data resiliency has a green tick indicating it is healthy. The RADOS Object Gateway is only listed in case RADOS Object Gateway endpoint details are included while deploying OpenShift Data Foundation in external mode.

Chapter 7. Adding annotation to the pre-existing backingstores

Adding the correct annotation to the pre-existing backingstores allows the backingstores backed by object gateways (RGWs) to report its actual and free size. The Multicloud Object Gateway (MCG) can retrieve and use this information. This flow is only relevant if RGW is present and in-use on the cluster. RGW is used by default only in on-premise platforms such as vSphere.

Note

If you added the annotations to pre-existing backingstores after upgrading to OpenShift Data Foundation version 4.8, then you do not need to add them after upgrading to 4.9. All backingstores created in version 4.8 and above will already have this annotation by default.

Procedure

  1. Log in to the OpenShift Container Platform Web Console.
  2. Click HomeSearch.
  3. Search for BackingStore in Resources and click on it.
  4. Beside the S3-compatible BackingStore, click Action Menu (⋮)Edit annotations.
  5. Add rgw for KEY.
  6. Click Save.

Chapter 8. Post-update configuration changes for clusters backed by local storage

In Red Hat OpenShift Container Platform 4.6 and onward, the Local Storage Operator provides new custom resource types for managing local storage:

  • LocalVolumeDiscovery
  • LocalVolumeSet

If you have incrementally upgraded to OpenShift Data Foundation version 4.9 from version 4.5 or earlier with Local Storage Operator, and these resources have not yet been created, additional configuration steps are required after an update to ensure that all features work as expected. These resource types are not automatically handled as part of an update from 4.5, and must be created manually. See Post-update configuration changes for clusters backed by local storage for instructions on creating the resources.

Note

If you had already created these resources after upgrading from 4.5, then you do not need to create them after upgrading to 4.9.

8.1. Adding annotations

You need to add annotations to the storage cluster to enable replacing of failed storage devices through the user interface when you upgraded to OpenShift Data Foundation version 4.9 from a previous version.

Procedure

  1. Log in to the OpenShift Container Platform Web Console.
  2. Click HomeSearch.
  3. Search for StorageCluster in Resources and click on it.
  4. Beside ocs-storagecluster, click Action menu (⋮)Edit annotations.
  5. Add cluster.ocs.openshift.io/local-devices and true for KEY and VALUE respectively.
  6. Click Save.