Chapter 15. Updating OpenShift Container Storage

Procedure

1. Log in to the OpenShift Container Platform cluster using the cluster-admin role.

2. Locate your auth.json file.

   This file is generated when you use podman or docker to log in to a registry. It is located in one of the following locations:

   • ~/.docker/auth.json
   • /run/user/<UID>/containers/auth.json
   • /var/run/containers/<UID>/auth.json

3. Obtain your unique Red Hat registry pull secret and paste it into your auth.json file. It will look something like this.

   {
       "auths": {
           "cloud.openshift.com": {
               "auth": "*****************",
               "email": "user@example.com"
           },
           "quay.io": {
               "auth": "*****************",
               "email": "user@example.com"
           },
           "registry.connect.redhat.com": {
               "auth": "*****************",
               "email": "user@example.com"
           },
           "registry.redhat.io": {
               "auth": "*****************",
               "email": "user@example.com"
           }
       }
     }

4. Export environment variables with the appropriate details for your setup.

   $ export AUTH_FILE="<location_of_auth.json>"
   $ export MIRROR_REGISTRY_DNS="<your_registry_url>:<port>"

5. Use podman to log in to the mirror registry and store the credentials in the ${AUTH_FILE}.

   $ podman login ${MIRROR_REGISTRY_DNS} --tls-verify=false --authfile ${AUTH_FILE}

   This adds the mirror registry to the auth.json file.

   {
       "auths": {
           "cloud.openshift.com": {
               "auth": "*****************",
               "email": "user@example.com"
           },
           "quay.io": {
               "auth": "*****************",
               "email": "user@example.com"
           },
           "registry.connect.redhat.com": {
               "auth": "*****************",
               "email": "user@example.com"
           },
           "registry.redhat.io": {
               "auth": "*****************",
               "email": "user@example.com"
           },
           "<mirror_registry>": {
               "auth": "*****************",
           }
       }
     }

Procedure

1. Build the catalog for redhat-operators.

   Set --from to the ose-operator-registry base image using the tag that matches the target OpenShift Container Platform cluster major and minor version.

   $ oc adm catalog build --appregistry-org redhat-operators \
     --from=registry.redhat.io/openshift4/ose-operator-registry:v4.6 \
     --to=${MIRROR_REGISTRY_DNS}/olm/redhat-operators:v2 \
     --registry-config=${AUTH_FILE} \
     --filter-by-os="linux/amd64" --insecure

2. Mirror the catalog for redhat-operators.

   This is a long operation and can take 1-5 hours. Make sure there is 100 GB available disk space on the mirror host.

   $ oc adm catalog mirror ${MIRROR_REGISTRY_DNS}/olm/redhat-operators:v2 \
   ${MIRROR_REGISTRY_DNS} --registry-config=${AUTH_FILE} --insecure

Procedure

1. Check the content of this file for the mirrors mapping shown as follows:

   spec:
     repositoryDigestMirrors:
       - mirrors:
         - <your_registry>/ocs4
         source: registry.redhat.io/ocs4
       - mirrors:
         - <your_registry>/rhceph
         source: registry.redhat.io/rhceph
       - mirrors:
         - <your_registry>/openshift4
         source: registry.redhat.io/openshift4
       - mirrors:
         - <your_registry>/rhscl
         source: registry.redhat.io/rhscl

2. Add any missing entries to the end of the imageContentSourcePolicy.yaml file.

3. Apply the imageContentSourcePolicy.yaml file to the cluster.

   $ oc apply -f ./[output dir]/imageContentSourcePolicy.yaml

   Once the Image Content Source Policy is updated, all the nodes (master, infra, and workers) in the cluster need to be updated and rebooted. This process is automatically handled through the Machine Config Pool operator and take up to 30 minutes although the exact elapsed time might vary based on the number of nodes in your OpenShift cluster. You can monitor the update process by using the oc get mcp command or the oc get node command.

Procedure

1. Recreate a CatalogSource object that references the catalog image for Red Hat operators.

   Note

   Make sure you have mirrored the correct catalog source with the correct version (that is, v2).

   Save the following in a redhat-operator-catalogsource.yaml file, remembering to replace <your_registry> with your mirror registry URL:

   apiVersion: operators.coreos.com/v1alpha1
   kind: CatalogSource
   metadata:
     name: redhat-operators
     namespace: openshift-marketplace
   spec:
     sourceType: grpc
     icon:
       base64data: PHN2ZyBpZD0iTGF5ZXJfMSIgZGF0YS1uYW1lPSJMYXllciAxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAxOTIgMTQ1Ij48ZGVmcz48c3R5bGU+LmNscy0xe2ZpbGw6I2UwMDt9PC9zdHlsZT48L2RlZnM+PHRpdGxlPlJlZEhhdC1Mb2dvLUhhdC1Db2xvcjwvdGl0bGU+PHBhdGggZD0iTTE1Ny43Nyw2Mi42MWExNCwxNCwwLDAsMSwuMzEsMy40MmMwLDE0Ljg4LTE4LjEsMTcuNDYtMzAuNjEsMTcuNDZDNzguODMsODMuNDksNDIuNTMsNTMuMjYsNDIuNTMsNDRhNi40Myw2LjQzLDAsMCwxLC4yMi0xLjk0bC0zLjY2LDkuMDZhMTguNDUsMTguNDUsMCwwLDAtMS41MSw3LjMzYzAsMTguMTEsNDEsNDUuNDgsODcuNzQsNDUuNDgsMjAuNjksMCwzNi40My03Ljc2LDM2LjQzLTIxLjc3LDAtMS4wOCwwLTEuOTQtMS43My0xMC4xM1oiLz48cGF0aCBjbGFzcz0iY2xzLTEiIGQ9Ik0xMjcuNDcsODMuNDljMTIuNTEsMCwzMC42MS0yLjU4LDMwLjYxLTE3LjQ2YTE0LDE0LDAsMCwwLS4zMS0zLjQybC03LjQ1LTMyLjM2Yy0xLjcyLTcuMTItMy4yMy0xMC4zNS0xNS43My0xNi42QzEyNC44OSw4LjY5LDEwMy43Ni41LDk3LjUxLjUsOTEuNjkuNSw5MCw4LDgzLjA2LDhjLTYuNjgsMC0xMS42NC01LjYtMTcuODktNS42LTYsMC05LjkxLDQuMDktMTIuOTMsMTIuNSwwLDAtOC40MSwyMy43Mi05LjQ5LDI3LjE2QTYuNDMsNi40MywwLDAsMCw0Mi41Myw0NGMwLDkuMjIsMzYuMywzOS40NSw4NC45NCwzOS40NU0xNjAsNzIuMDdjMS43Myw4LjE5LDEuNzMsOS4wNSwxLjczLDEwLjEzLDAsMTQtMTUuNzQsMjEuNzctMzYuNDMsMjEuNzdDNzguNTQsMTA0LDM3LjU4LDc2LjYsMzcuNTgsNTguNDlhMTguNDUsMTguNDUsMCwwLDEsMS41MS03LjMzQzIyLjI3LDUyLC41LDU1LC41LDc0LjIyYzAsMzEuNDgsNzQuNTksNzAuMjgsMTMzLjY1LDcwLjI4LDQ1LjI4LDAsNTYuNy0yMC40OCw1Ni43LTM2LjY1LDAtMTIuNzItMTEtMjcuMTYtMzAuODMtMzUuNzgiLz48L3N2Zz4=
       mediatype: image/svg+xml
     image: <your_registry>/olm/redhat-operators:v2
     displayName: Redhat Operators Catalog
     publisher: Red Hat

2. Create a catalogsource using the redhat-operator-catalogsource.yaml file:

   $ oc apply -f redhat-operator-catalogsource.yaml

3. Verify that the new redhat-operator pod is running.

   $ oc get pod -n openshift-marketplace | grep redhat-operators

Procedure

1. Log in to OpenShift Web Console.
2. Click Operators → Installed Operators
3. Select the openshift-storage project.
4. Click the OpenShift Container Storage operator name.
5. Click the Subscription tab and click the link under Approval.
6. Select Automatic (default) and click Save.

7. Perform one of the following depending on the Upgrade Status:

   • Upgrade Status shows requires approval.

     Note

     Upgrade status shows requires approval if the new OpenShift Container Storage version is already detected in the channel, and approval strategy was changed from Manual to Automatic at the time of update.

     1. Click on the Install Plan link.
     2. On the InstallPlan Details page, click Preview Install Plan.
     3. Review the install plan and click Approve.
     4. Wait for the Status to change from Unknown to Created.
     5. Click Operators → Installed Operators
     6. Select the openshift-storage project.
     7. Wait for the Status to change to Up to date

   • Upgrade Status does not show requires approval:

     1. Wait for the update to initiate. This may take up to 20 minutes.
     2. Click Operators → Installed Operators
     3. Select the openshift-storage project.
     4. Wait for the Status to change to Up to date

Verification steps

1. Click Overview → Persistent Storage tab and in the Status card confirm that the OCS Cluster and Data Resiliency has a green tick mark indicating it is healthy.
2. Click Overview → Object Service tab and in the Status card confirm that both the Object Service and Data Resiliency are in Ready state (green tick) indicating it is healthy.

3. Click Operators → Installed Operators → OpenShift Container Storage Operator. Under Storage Cluster, verify that the cluster service status is Ready.

   Note

   Once updated from OpenShift Container Storage version 4.5 to 4.6, the Version field here will still display 4.5. This is because the ocs-operator does not update the string represented in this field.

4. Ensure that all OpenShift Container Storage Pods, including the operator pods, are in Running state in the openshift-storage namespace.

   To view the state of the pods, click Workloads → Pods. Select openshift-storage from the Project drop down list.

5. If verification steps fail, contact Red Hat Support.

Procedure

1. Log in to OpenShift Web Console.
2. Click Operators → Installed Operators
3. Select the openshift-storage project.
4. Click the OpenShift Container Storage operator name.
5. Click the Subscription tab and click the link under Approval.
6. Select Manual and click Save.
7. Wait for the Upgrade Status to change to Upgrading.
8. If the Upgrade Status shows requires approval, click on requires approval.
9. On the InstallPlan Details page, click Preview Install Plan.
10. Review the install plan and click Approve.
11. Wait for the Status to change from Unknown to Created.
12. Click Operators → Installed Operators
13. Select the openshift-storage project.
14. Wait for the Status to change to Up to date

Verification steps

1. Click Overview → Persistent Storage tab and in the Status card confirm that the OCS Cluster and Data Resiliency has a green tick mark indicating it is healthy.
2. Click Overview → Object Service tab and in the Status card confirm that both the Object Service and Data Resiliency are in Ready state (green tick) indicating it is healthy.

3. Click Operators → Installed Operators → OpenShift Container Storage Operator. Under Storage Cluster, verify that the cluster service status is Ready.

   Note

   Once updated from OpenShift Container Storage version 4.5 to 4.6, the Version field here will still display 4.5. This is because the ocs-operator does not update the string represented in this field.

4. Ensure that all OpenShift Container Storage Pods, including the operator pods, are in Running state in the openshift-storage namespace.

   To view the state of the pods, click Workloads → Pods from the left pane of the OpenShift Web Console. Select openshift-storage from the Project drop down list.

5. If verification steps fail, contact Red Hat Support.