Chapter 13. Updating a cluster in a disconnected environment

Procedure

1. Navigate to the OpenShift Container Platform downloads page on the Red Hat Customer Portal.
2. Select the architecture from the Product Variant drop-down list.
3. Select the appropriate version from the Version drop-down list.
4. Click Download Now next to the OpenShift v4.12 Linux Client entry and save the file.

5. Unpack the archive:

   $ tar xvf <file>

6. Place the oc binary in a directory that is on your PATH.

   To check your PATH, execute the following command:

   $ echo $PATH

Procedure

1. Navigate to the OpenShift Container Platform downloads page on the Red Hat Customer Portal.
2. Select the appropriate version from the Version drop-down list.
3. Click Download Now next to the OpenShift v4.12 Windows Client entry and save the file.
4. Unzip the archive with a ZIP program.

5. Move the oc binary to a directory that is on your PATH.

   To check your PATH, open the command prompt and execute the following command:

   C:\> path

Procedure

1. Navigate to the OpenShift Container Platform downloads page on the Red Hat Customer Portal.
2. Select the appropriate version from the Version drop-down list.

3. Click Download Now next to the OpenShift v4.12 macOS Client entry and save the file.

   Note

   For macOS arm64, choose the OpenShift v4.12 macOS arm64 Client entry.

4. Unpack and unzip the archive.

5. Move the oc binary to a directory on your PATH.

   To check your PATH, open a terminal and execute the following command:

   $ echo $PATH

1. Download your registry.redhat.io pull secret from the Red Hat OpenShift Cluster Manager.

2. Make a copy of your pull secret in JSON format:

   $ cat ./pull-secret | jq . > <path>/<pull_secret_file_in_json> 1

   1

   Specify the path to the folder to store the pull secret in and a name for the JSON file that you create.

   The contents of the file resemble the following example:

   {
     "auths": {
       "cloud.openshift.com": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "quay.io": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "registry.connect.redhat.com": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       },
       "registry.redhat.io": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       }
     }
   }

3. Optional: If using the oc-mirror plugin, save the file either as ~/.docker/config.json or $XDG_RUNTIME_DIR/containers/auth.json.

4. Generate the base64-encoded user name and password or token for your mirror registry:

   $ echo -n '<user_name>:<password>' | base64 -w0 1
   BGVtbYk3ZHAtqXs=

   1

   For <user_name> and <password>, specify the user name and password that you configured for your registry.

5. Edit the JSON file and add a section that describes your registry to it:

     "auths": {
       "<mirror_registry>": { 1
         "auth": "<credentials>", 2
         "email": "you@example.com"
       }
     },

   1

   For <mirror_registry>, specify the registry domain name, and optionally the port, that your mirror registry uses to serve content. For example, registry.example.com or registry.example.com:8443

   2

   For <credentials>, specify the base64-encoded user name and password for the mirror registry.

   The file resembles the following example:

   {
     "auths": {
       "registry.example.com": {
         "auth": "BGVtbYk3ZHAtqXs=",
         "email": "you@example.com"
       },
       "cloud.openshift.com": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "quay.io": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "registry.connect.redhat.com": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       },
       "registry.redhat.io": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       }
     }
   }

Procedure

1. Download the oc-mirror CLI plugin.

   1. Navigate to the Downloads page of the OpenShift Cluster Manager Hybrid Cloud Console.
   2. Under the OpenShift disconnected installation tools section, click Download for OpenShift Client (oc) mirror plugin and save the file.

2. Extract the archive:

   $ tar xvzf oc-mirror.tar.gz

3. If necessary, update the plugin file to be executable:

   $ chmod +x oc-mirror

   Note

   Do not rename the oc-mirror file.

4. Install the oc-mirror CLI plugin by placing the file in your PATH, for example, /usr/local/bin:

   $ sudo mv oc-mirror /usr/local/bin/.

Procedure

1. Use the oc mirror init command to create a template for the image set configuration and save it to a file called imageset-config.yaml:

   $ oc mirror init --registry example.com/mirror/oc-mirror-metadata > imageset-config.yaml 1

   1

   Replace example.com/mirror/oc-mirror-metadata with the location of your registry for the storage backend.

2. Edit the file and adjust the settings as necessary:

   kind: ImageSetConfiguration
   apiVersion: mirror.openshift.io/v1alpha2
   archiveSize: 4                                                      1
   storageConfig:                                                      2
     registry:
       imageURL: example.com/mirror/oc-mirror-metadata                 3
       skipTLS: false
   mirror:
     platform:
       channels:
       - name: stable-4.12                                             4
         type: ocp
       graph: true                                                     5
     operators:
     - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.12  6
       packages:
       - name: serverless-operator                                     7
         channels:
         - name: stable                                                8
     additionalImages:
     - name: registry.redhat.io/ubi8/ubi:latest                        9
     helm: {}

   1

   Add archiveSize to set the maximum size, in GiB, of each file within the image set.

   2

   Set the back-end location to save the image set metadata to. This location can be a registry or local directory. It is required to specify storageConfig values, unless you are using the Technology Preview OCI feature.

   3

   Set the registry URL for the storage backend.

   4

   Set the channel to retrieve the OpenShift Container Platform images from.

   5

   Add graph: true to build and push the graph-data image to the mirror registry. The graph-data image is required to create OpenShift Update Service (OSUS). The graph: true field also generates the UpdateService custom resource manifest. The oc command-line interface (CLI) can use the UpdateService custom resource manifest to create OSUS. For more information, see About the OpenShift Update Service.

   6

   Set the Operator catalog to retrieve the OpenShift Container Platform images from.

   7

   Specify only certain Operator packages to include in the image set. Remove this field to retrieve all packages in the catalog.

   8

   Specify only certain channels of the Operator packages to include in the image set. You must always include the default channel for the Operator package even if you do not use the bundles in that channel. You can find the default channel by running the following command: oc mirror list operators --catalog=<catalog_name> --package=<package_name>.

   9

   Specify any additional images to include in image set.

   Note

   The graph: true field also mirrors the ubi-micro image along with other mirrored images.

   See Image set configuration parameters for the full list of parameters and Image set configuration examples for various mirroring use cases.

3. Save the updated file.

   This image set configuration file is required by the oc mirror command when mirroring content.

Procedure

1. Log in to the OpenShift CLI as a user with the cluster-admin role.

2. Apply the YAML files from the results directory to the cluster by running the following command:

   $ oc apply -f ./oc-mirror-workspace/results-1639608409/

3. If you mirrored release images, apply the release image signatures to the cluster by running the following command:

   $ oc apply -f ./oc-mirror-workspace/results-1639608409/release-signatures/

   Note

   If you are mirroring Operators instead of clusters, you do not need to run $ oc apply -f ./oc-mirror-workspace/results-1639608409/release-signatures/. Running that command will return an error, as there are no release image signatures to apply.

Verification

1. Verify that the ImageContentSourcePolicy resources were successfully installed by running the following command:

   $ oc get imagecontentsourcepolicy

2. Verify that the CatalogSource resources were successfully installed by running the following command:

   $ oc get catalogsource -n openshift-marketplace

Procedure

1. Run the oc mirror command with the --dry-run flag to perform a dry run:

   $ oc mirror --config=./imageset-config.yaml \1
     docker://registry.example:5000            \2
     --dry-run                                  3

   1

   Pass in the image set configuration file that was created. This procedure assumes that it is named imageset-config.yaml.

   2

   Specify the mirror registry. Nothing is mirrored to this registry as long as you use the --dry-run flag.

   3

   Use the --dry-run flag to generate the dry run artifacts and not an actual image set file.

   Example output

   Checking push permissions for registry.example:5000
   Creating directory: oc-mirror-workspace/src/publish
   Creating directory: oc-mirror-workspace/src/v2
   Creating directory: oc-mirror-workspace/src/charts
   Creating directory: oc-mirror-workspace/src/release-signatures
   No metadata detected, creating new workspace
   wrote mirroring manifests to oc-mirror-workspace/operators.1658342351/manifests-redhat-operator-index
   
   ...
   
   info: Planning completed in 31.48s
   info: Dry run complete
   Writing image mapping to oc-mirror-workspace/mapping.txt

2. Navigate into the workspace directory that was generated:

   $ cd oc-mirror-workspace/

3. Review the mapping.txt file that was generated.

   This file contains a list of all images that would be mirrored.

4. Review the pruning-plan.json file that was generated.

   This file contains a list of all images that would be pruned from the mirror registry when the image set is published.

   Note

   The pruning-plan.json file is only generated if your oc-mirror command points to your mirror registry and there are images to be pruned.

Procedure

1. Optional: Retrieve the catalogs and images that you require and save them to disk. If you already have the catalog image in OCI format on disk, you can skip this step.

   1. Create the image set configuration file:

      Example image set configuration file for copying to disk

      kind: ImageSetConfiguration
      apiVersion: mirror.openshift.io/v1alpha2
      mirror:
       operators:
       - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.12
         packages:
         - name: aws-load-balancer-operator

      Note

      When using the OCI feature, only the mirror.operators.catalog setting is available for use.

      The storageConfig setting is ignored in favor of the location passed in to the oc mirror command.

   2. Run the oc mirror command to mirror the images from the specified image set configuration to disk:

      $ oc mirror --config=./imageset-config.yaml \ 1
        --use-oci-feature \                         2
        --oci-feature-action=copy \                 3
        oci://my-oci-catalog                        4

      1

      Pass in the image set configuration file. This procedure assumes that it is named imageset-config.yaml.

      2

      Use the --use-oci-feature flag to enable the OCI feature.

      3

      To copy the catalog to disk, set the --oci-feature-action flag to copy.

      4

      Specify a directory on disk where you want to output the catalog. This procedure assumes that it is named my-oci-catalog. The path must start with oci://. If the specified directory is not a full path, the directory is created in the current working directory where the oc mirror command is run.

      Note

      You can optionally use the --oci-registries-config flag to specify the path to a TOML-formatted registries.conf file. You can use this to mirror from a different registry, such as a pre-production location for testing, without having to change the image set configuration file.

      Example registries.conf file

      [[registry]]
       location = "registry.redhat.io:5000"
       insecure = false
       blocked = false
       mirror-by-digest-only = true
       prefix = ""
       [[registry.mirror]]
          location = "preprod-registry.example.com"
          insecure = false

      Set the location field in the registry.mirror section to an alternative registry location that you want to pull images from. The location field in the registry section must be the same registry location as the one you specify in the image set configuration file.

   3. List your directory contents and verify that the following directories were created:

      $ ls -l

      Example output

      my-oci-catalog      1
      oc-mirror-workspace 2
      olm_artifacts       3

      1

      Directory that contains the OCI catalog. This procedure assumes that it is named my-oci-catalog.

      2

      Directory that contains each image in the catalog in its original format.

      3

      Directory that contains the files that describe the Operator bundles that this catalog references.

2. Update the image set configuration file to specify the location of the catalog on disk to mirror to the target mirror registry:

   Example image set configuration file for mirroring to mirror registry

   kind: ImageSetConfiguration
   apiVersion: mirror.openshift.io/v1alpha2
   mirror:
    operators:
    - catalog: oci:///home/user/oc-mirror/my-oci-catalog/redhat-operator-index 1
      packages:
      - name: aws-load-balancer-operator

   1

   Specify the absolute path to the location of the OCI catalog on disk. This procedure assumes that you used my-oci-catalog as the directory and mirrored the redhat-operator-index catalog. The path must start with oci://.

3. Run the oc mirror command to process the image set file on disk and mirror the contents to a target mirror registry:

   $ oc mirror --config=./imageset-config.yaml \ 1
     --use-oci-feature \                         2
     --oci-feature-action=mirror \               3
     docker://registry.example:5000              4

   1

   Pass in the updated image set configuration file. This procedure assumes that it is named imageset-config.yaml.

   2

   Use the --use-oci-feature flag to enable the OCI feature.

   3

   To mirror the catalog to the target mirror registry, set the --oci-feature-action flag to mirror.

   4

   Specify the registry to mirror the image set file to. The registry must start with docker://. If you specify a top-level namespace for the mirror registry, you must also use this same namespace on subsequent executions.

   Note

   You can optionally use the --oci-insecure-signature-policy flag to not push signatures to the target mirror registry.

Procedure

1. In the web console, click Operators → OperatorHub.

   Note

   Enter Update Service into the Filter by keyword…​ field to find the Operator faster.

2. Choose OpenShift Update Service from the list of available Operators, and click Install.

   1. Channel v1 is selected as the Update Channel since it is the only channel available in this release.
   2. Select A specific namespace on the cluster under Installation Mode.
   3. Select a namespace for Installed Namespace or accept the recommended namespace openshift-update-service.

   4. Select an Approval Strategy:

      • The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.
      • The Manual strategy requires a cluster administrator to approve the Operator update.
   5. Click Install.
3. Verify that the OpenShift Update Service Operator is installed by switching to the Operators → Installed Operators page.
4. Ensure that OpenShift Update Service is listed in the selected namespace with a Status of Succeeded.

Procedure

1. Create a namespace for the OpenShift Update Service Operator:

   1. Create a Namespace object YAML file, for example, update-service-namespace.yaml, for the OpenShift Update Service Operator:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-update-service
        annotations:
          openshift.io/node-selector: ""
        labels:
          openshift.io/cluster-monitoring: "true" 1

      1

      Set the openshift.io/cluster-monitoring label to enable Operator-recommended cluster monitoring on this namespace.

   2. Create the namespace:

      $ oc create -f <filename>.yaml

      For example:

      $ oc create -f update-service-namespace.yaml

2. Install the OpenShift Update Service Operator by creating the following objects:

   1. Create an OperatorGroup object YAML file, for example, update-service-operator-group.yaml:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: update-service-operator-group
      spec:
        targetNamespaces:
        - openshift-update-service

   2. Create an OperatorGroup object:

      $ oc -n openshift-update-service create -f <filename>.yaml

      For example:

      $ oc -n openshift-update-service create -f update-service-operator-group.yaml

   3. Create a Subscription object YAML file, for example, update-service-subscription.yaml:

      Example Subscription

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: update-service-subscription
      spec:
        channel: v1
        installPlanApproval: "Automatic"
        source: "redhat-operators" 1
        sourceNamespace: "openshift-marketplace"
        name: "cincinnati-operator"

      1

      Specify the name of the catalog source that provides the Operator. For clusters that do not use a custom Operator Lifecycle Manager (OLM), specify redhat-operators. If your OpenShift Container Platform cluster is installed in a disconnected environment, specify the name of the CatalogSource object created when you configured Operator Lifecycle Manager (OLM).

   4. Create the Subscription object:

      $ oc create -f <filename>.yaml

      For example:

      $ oc -n openshift-update-service create -f update-service-subscription.yaml

      The OpenShift Update Service Operator is installed to the openshift-update-service namespace and targets the openshift-update-service namespace.

3. Verify the Operator installation:

   $ oc -n openshift-update-service get clusterserviceversions

   Example output

   NAME                             DISPLAY                    VERSION   REPLACES   PHASE
   update-service-operator.v4.6.0   OpenShift Update Service   4.6.0                Succeeded
   ...

   If the OpenShift Update Service Operator is listed, the installation was successful. The version number might be different than shown.

Procedure

1. In the web console, click Operators → Installed Operators.
2. Choose OpenShift Update Service from the list of installed Operators.
3. Click the Update Service tab.
4. Click Create UpdateService.
5. Enter a name in the Name field, for example, service.
6. Enter the local pullspec in the Graph Data Image field to the graph data container image created in "Creating the OpenShift Update Service graph data container image", for example, registry.example.com/openshift/graph-data:latest.
7. In the Releases field, enter the local registry and repository created to contain the release images in "Mirroring the OpenShift Container Platform image repository", for example, registry.example.com/ocp4/openshift4-release-images.
8. Enter 2 in the Replicas field.
9. Click Create to create the OpenShift Update Service application.

10. Verify the OpenShift Update Service application:

    • From the UpdateServices list in the Update Service tab, click the Update Service application just created.
    • Click the Resources tab.
    • Verify each application resource has a status of Created.

Procedure

1. Configure the OpenShift Update Service target namespace, for example, openshift-update-service:

   $ NAMESPACE=openshift-update-service

   The namespace must match the targetNamespaces value from the operator group.

2. Configure the name of the OpenShift Update Service application, for example, service:

   $ NAME=service

3. Configure the local registry and repository for the release images as configured in "Mirroring the OpenShift Container Platform image repository", for example, registry.example.com/ocp4/openshift4-release-images:

   $ RELEASE_IMAGES=registry.example.com/ocp4/openshift4-release-images

4. Set the local pullspec for the graph data image to the graph data container image created in "Creating the OpenShift Update Service graph data container image", for example, registry.example.com/openshift/graph-data:latest:

   $ GRAPH_DATA_IMAGE=registry.example.com/openshift/graph-data:latest

5. Create an OpenShift Update Service application object:

   $ oc -n "${NAMESPACE}" create -f - <<EOF
   apiVersion: updateservice.operator.openshift.io/v1
   kind: UpdateService
   metadata:
     name: ${NAME}
   spec:
     replicas: 2
     releases: ${RELEASE_IMAGES}
     graphDataImage: ${GRAPH_DATA_IMAGE}
   EOF

6. Verify the OpenShift Update Service application:

   1. Use the following command to obtain a policy engine route:

      $ while sleep 1; do POLICY_ENGINE_GRAPH_URI="$(oc -n "${NAMESPACE}" get -o jsonpath='{.status.policyEngineURI}/api/upgrades_info/v1/graph{"\n"}' updateservice "${NAME}")"; SCHEME="${POLICY_ENGINE_GRAPH_URI%%:*}"; if test "${SCHEME}" = http -o "${SCHEME}" = https; then break; fi; done

      You might need to poll until the command succeeds.

   2. Retrieve a graph from the policy engine. Be sure to specify a valid version for channel. For example, if running in OpenShift Container Platform 4.12, use stable-4.12:

      $ while sleep 10; do HTTP_CODE="$(curl --header Accept:application/json --output /dev/stderr --write-out "%{http_code}" "${POLICY_ENGINE_GRAPH_URI}?channel=stable-4.6")"; if test "${HTTP_CODE}" -eq 200; then break; fi; echo "${HTTP_CODE}"; done

      This polls until the graph request succeeds; however, the resulting graph might be empty depending on which release images you have mirrored.

Procedure

1. In the web console, click Operators → Installed Operators.
2. Choose OpenShift Update Service from the list of installed Operators.
3. Click the Update Service tab.
4. From the list of installed OpenShift Update Service applications, select the application to be deleted and then click Delete UpdateService.
5. From the Delete UpdateService? confirmation dialog, click Delete to confirm the deletion.

Procedure

1. Get the OpenShift Update Service application name using the namespace the OpenShift Update Service application was created in, for example, openshift-update-service:

   $ oc get updateservice -n openshift-update-service

   Example output

   NAME      AGE
   service   6s

2. Delete the OpenShift Update Service application using the NAME value from the previous step and the namespace the OpenShift Update Service application was created in, for example, openshift-update-service:

   $ oc delete updateservice service -n openshift-update-service

   Example output

   updateservice.updateservice.operator.openshift.io "service" deleted

Procedure

1. In the web console, click Operators → Installed Operators.
2. Select OpenShift Update Service from the list of installed Operators and click Uninstall Operator.
3. From the Uninstall Operator? confirmation dialog, click Uninstall to confirm the uninstallation.

Procedure

1. Change to the project containing the OpenShift Update Service Operator, for example, openshift-update-service:

   $ oc project openshift-update-service

   Example output

   Now using project "openshift-update-service" on server "https://example.com:6443".

2. Get the name of the OpenShift Update Service Operator operator group:

   $ oc get operatorgroup

   Example output

   NAME                             AGE
   openshift-update-service-fprx2   4m41s

3. Delete the operator group, for example, openshift-update-service-fprx2:

   $ oc delete operatorgroup openshift-update-service-fprx2

   Example output

   operatorgroup.operators.coreos.com "openshift-update-service-fprx2" deleted

4. Get the name of the OpenShift Update Service Operator subscription:

   $ oc get subscription

   Example output

   NAME                      PACKAGE                   SOURCE                        CHANNEL
   update-service-operator   update-service-operator   updateservice-index-catalog   v1

5. Using the Name value from the previous step, check the current version of the subscribed OpenShift Update Service Operator in the currentCSV field:

   $ oc get subscription update-service-operator -o yaml | grep " currentCSV"

   Example output

     currentCSV: update-service-operator.v0.0.1

6. Delete the subscription, for example, update-service-operator:

   $ oc delete subscription update-service-operator

   Example output

   subscription.operators.coreos.com "update-service-operator" deleted

7. Delete the CSV for the OpenShift Update Service Operator using the currentCSV value from the previous step:

   $ oc delete clusterserviceversion update-service-operator.v0.0.1

   Example output

   clusterserviceversion.operators.coreos.com "update-service-operator.v0.0.1" deleted