Chapter 14. Updating a cluster in a disconnected environment

Procedure

1. Use the Red Hat OpenShift Container Platform Upgrade Graph visualizer and update planner to plan an update from one version to another. The OpenShift Upgrade Graph provides channel graphs and a way to confirm that there is an update path between your current and intended cluster versions.

2. Set the required environment variables:

   1. Export the release version:

      $ export OCP_RELEASE=<release_version>

      For <release_version>, specify the tag that corresponds to the version of OpenShift Container Platform to which you want to update, such as 4.5.4.

   2. Export the local registry name and host port:

      $ LOCAL_REGISTRY='<local_registry_host_name>:<local_registry_host_port>'

      For <local_registry_host_name>, specify the registry domain name for your mirror repository, and for <local_registry_host_port>, specify the port that it serves content on.

   3. Export the local repository name:

      $ LOCAL_REPOSITORY='<local_repository_name>'

      For <local_repository_name>, specify the name of the repository to create in your registry, such as ocp4/openshift4.

   4. If you are using the OpenShift Update Service, export an additional local repository name to contain the release images:

      $ LOCAL_RELEASE_IMAGES_REPOSITORY='<local_release_images_repository_name>'

      For <local_release_images_repository_name>, specify the name of the repository to create in your registry, such as ocp4/openshift4-release-images.

   5. Export the name of the repository to mirror:

      $ PRODUCT_REPO='openshift-release-dev'

      For a production release, you must specify openshift-release-dev.

   6. Export the path to your registry pull secret:

      $ LOCAL_SECRET_JSON='<path_to_pull_secret>'

      For <path_to_pull_secret>, specify the absolute path to and file name of the pull secret for your mirror registry that you created.

      Note

      If your cluster uses an ImageContentSourcePolicy object to configure repository mirroring, you can use only global pull secrets for mirrored registries. You cannot add a pull secret to a project.

   7. Export the release mirror:

      $ RELEASE_NAME="ocp-release"

      For a production release, you must specify ocp-release.

   8. Export the type of architecture for your cluster:

      $ ARCHITECTURE=<cluster_architecture> 1

      1

      Specify the architecture of the cluster, such as x86_64, aarch64, s390x, or ppc64le.

   9. Export the path to the directory to host the mirrored images:

      $ REMOVABLE_MEDIA_PATH=<path> 1

      1

      Specify the full path, including the initial forward slash (/) character.

3. Review the images and configuration manifests to mirror:

   $ oc adm release mirror -a ${LOCAL_SECRET_JSON} --to-dir=${REMOVABLE_MEDIA_PATH}/mirror quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} --dry-run

4. Mirror the version images to the mirror registry.

   • If your mirror host does not have internet access, take the following actions:

     1. Connect the removable media to a system that is connected to the internet.

     2. Mirror the images and configuration manifests to a directory on the removable media:

        $ oc adm release mirror -a ${LOCAL_SECRET_JSON} --to-dir=${REMOVABLE_MEDIA_PATH}/mirror quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE}

        Note

        This command also generates and saves the mirrored release image signature config map onto the removable media.

     3. Take the media to the disconnected environment and upload the images to the local container registry.

        $ oc image mirror  -a ${LOCAL_SECRET_JSON} --from-dir=${REMOVABLE_MEDIA_PATH}/mirror "file://openshift/release:${OCP_RELEASE}*" ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} 1

        1

        For REMOVABLE_MEDIA_PATH, you must use the same path that you specified when you mirrored the images.

     4. Use oc command-line interface (CLI) to log in to the cluster that you are upgrading.

     5. Apply the mirrored release image signature config map to the connected cluster:

        $ oc apply -f ${REMOVABLE_MEDIA_PATH}/mirror/config/<image_signature_file> 1

        1

        For <image_signature_file>, specify the path and name of the file, for example, signature-sha256-81154f5c03294534.yaml.

     6. If you are using the OpenShift Update Service, mirror the release image to a separate repository:

        $ oc image mirror -a ${LOCAL_SECRET_JSON} ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE} ${LOCAL_REGISTRY}/${LOCAL_RELEASE_IMAGES_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}

   • If the local container registry and the cluster are connected to the mirror host, take the following actions:

     1. Directly push the release images to the local registry and apply the config map to the cluster by using following command:

        $ oc adm release mirror -a ${LOCAL_SECRET_JSON} --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
          --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} --apply-release-image-signature

        Note

        If you include the --apply-release-image-signature option, do not create the config map for image signature verification.

     2. If you are using the OpenShift Update Service, mirror the release image to a separate repository:

        $ oc image mirror -a ${LOCAL_SECRET_JSON} ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE} ${LOCAL_REGISTRY}/${LOCAL_RELEASE_IMAGES_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}

Procedure

1. In the web console, click Operators → OperatorHub.

   Note

   Enter Update Service into the Filter by keyword…​ field to find the Operator faster.

2. Choose OpenShift Update Service from the list of available Operators, and click Install.

   1. Channel v1 is selected as the Update Channel since it is the only channel available in this release.
   2. Select A specific namespace on the cluster under Installation Mode.
   3. Select a namespace for Installed Namespace or accept the recommended namespace openshift-update-service.

   4. Select an Approval Strategy:

      • The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.
      • The Manual strategy requires a cluster administrator to approve the Operator update.
   5. Click Install.
3. Verify that the OpenShift Update Service Operator is installed by switching to the Operators → Installed Operators page.
4. Ensure that OpenShift Update Service is listed in the selected namespace with a Status of Succeeded.

Procedure

1. Create a namespace for the OpenShift Update Service Operator:

   1. Create a Namespace object YAML file, for example, update-service-namespace.yaml, for the OpenShift Update Service Operator:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-update-service
        annotations:
          openshift.io/node-selector: ""
        labels:
          openshift.io/cluster-monitoring: "true" 1

      1

      Set the openshift.io/cluster-monitoring label to enable Operator-recommended cluster monitoring on this namespace.

   2. Create the namespace:

      $ oc create -f <filename>.yaml

      For example:

      $ oc create -f update-service-namespace.yaml

2. Install the OpenShift Update Service Operator by creating the following objects:

   1. Create an OperatorGroup object YAML file, for example, update-service-operator-group.yaml:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: update-service-operator-group
      spec:
        targetNamespaces:
        - openshift-update-service

   2. Create an OperatorGroup object:

      $ oc -n openshift-update-service create -f <filename>.yaml

      For example:

      $ oc -n openshift-update-service create -f update-service-operator-group.yaml

   3. Create a Subscription object YAML file, for example, update-service-subscription.yaml:

      Example Subscription

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: update-service-subscription
      spec:
        channel: v1
        installPlanApproval: "Automatic"
        source: "redhat-operators" 1
        sourceNamespace: "openshift-marketplace"
        name: "cincinnati-operator"

      1

      Specify the name of the catalog source that provides the Operator. For clusters that do not use a custom Operator Lifecycle Manager (OLM), specify redhat-operators. If your OpenShift Container Platform cluster is installed in a disconnected environment, specify the name of the CatalogSource object created when you configured Operator Lifecycle Manager (OLM).

   4. Create the Subscription object:

      $ oc create -f <filename>.yaml

      For example:

      $ oc -n openshift-update-service create -f update-service-subscription.yaml

      The OpenShift Update Service Operator is installed to the openshift-update-service namespace and targets the openshift-update-service namespace.

3. Verify the Operator installation:

   $ oc -n openshift-update-service get clusterserviceversions

   Example output

   NAME                             DISPLAY                    VERSION   REPLACES   PHASE
   update-service-operator.v4.6.0   OpenShift Update Service   4.6.0                Succeeded
   ...

   If the OpenShift Update Service Operator is listed, the installation was successful. The version number might be different than shown.

Procedure

1. In the web console, click Operators → Installed Operators.
2. Choose OpenShift Update Service from the list of installed Operators.
3. Click the Update Service tab.
4. Click Create UpdateService.
5. Enter a name in the Name field, for example, service.
6. Enter the local pullspec in the Graph Data Image field to the graph data container image created in "Creating the OpenShift Update Service graph data container image", for example, registry.example.com/openshift/graph-data:latest.
7. In the Releases field, enter the local registry and repository created to contain the release images in "Mirroring the OpenShift Container Platform image repository", for example, registry.example.com/ocp4/openshift4-release-images.
8. Enter 2 in the Replicas field.
9. Click Create to create the OpenShift Update Service application.

10. Verify the OpenShift Update Service application:

    • From the UpdateServices list in the Update Service tab, click the Update Service application just created.
    • Click the Resources tab.
    • Verify each application resource has a status of Created.

Procedure

1. Configure the OpenShift Update Service target namespace, for example, openshift-update-service:

   $ NAMESPACE=openshift-update-service

   The namespace must match the targetNamespaces value from the operator group.

2. Configure the name of the OpenShift Update Service application, for example, service:

   $ NAME=service

3. Configure the local registry and repository for the release images as configured in "Mirroring the OpenShift Container Platform image repository", for example, registry.example.com/ocp4/openshift4-release-images:

   $ RELEASE_IMAGES=registry.example.com/ocp4/openshift4-release-images

4. Set the local pullspec for the graph data image to the graph data container image created in "Creating the OpenShift Update Service graph data container image", for example, registry.example.com/openshift/graph-data:latest:

   $ GRAPH_DATA_IMAGE=registry.example.com/openshift/graph-data:latest

5. Create an OpenShift Update Service application object:

   $ oc -n "${NAMESPACE}" create -f - <<EOF
   apiVersion: updateservice.operator.openshift.io/v1
   kind: UpdateService
   metadata:
     name: ${NAME}
   spec:
     replicas: 2
     releases: ${RELEASE_IMAGES}
     graphDataImage: ${GRAPH_DATA_IMAGE}
   EOF

6. Verify the OpenShift Update Service application:

   1. Use the following command to obtain a policy engine route:

      $ while sleep 1; do POLICY_ENGINE_GRAPH_URI="$(oc -n "${NAMESPACE}" get -o jsonpath='{.status.policyEngineURI}/api/upgrades_info/v1/graph{"\n"}' updateservice "${NAME}")"; SCHEME="${POLICY_ENGINE_GRAPH_URI%%:*}"; if test "${SCHEME}" = http -o "${SCHEME}" = https; then break; fi; done

      You might need to poll until the command succeeds.

   2. Retrieve a graph from the policy engine. Be sure to specify a valid version for channel. For example, if running in OpenShift Container Platform 4.13, use stable-4.13:

      $ while sleep 10; do HTTP_CODE="$(curl --header Accept:application/json --output /dev/stderr --write-out "%{http_code}" "${POLICY_ENGINE_GRAPH_URI}?channel=stable-4.6")"; if test "${HTTP_CODE}" -eq 200; then break; fi; echo "${HTTP_CODE}"; done

      This polls until the graph request succeeds; however, the resulting graph might be empty depending on which release images you have mirrored.

Procedure

1. Use the following command to convert one or more ImageContentSourcePolicy YAML files to an ImageDigestMirrorSet YAML file:

   $ oc adm migrate icsp <file_name>.yaml <file_name>.yaml <file_name>.yaml --dest-dir <path_to_the_directory>

   where:

   <file_name>

   Specifies the name of the source ImageContentSourcePolicy YAML. You can list multiple file names.

   --dest-dir

   Optional: Specifies a directory for the output ImageDigestMirrorSet YAML. If unset, the file is written to the current directory.

   For example, the following command converts the icsp.yaml and icsp-2.yaml file and saves the new YAML files to the idms-files directory.

   $ oc adm migrate icsp icsp.yaml icsp-2.yaml --dest-dir idms-files

   Example output

   wrote ImageDigestMirrorSet to idms-files/imagedigestmirrorset_ubi8repo.5911620242173376087.yaml
   wrote ImageDigestMirrorSet to idms-files/imagedigestmirrorset_ubi9repo.6456931852378115011.yaml

2. Create the CR object by running the following command:

   $ oc create -f <path_to_the_directory>/<file-name>.yaml

   where:

   <path_to_the_directory>

   Specifies the path to the directory, if you used the --dest-dir flag.

   <file_name>

   Specifies the name of the ImageDigestMirrorSet YAML.

3. Remove the ICSP objects after the IDMS objects are rolled out.

Procedure

1. In the web console, click Operators → Installed Operators.
2. Choose OpenShift Update Service from the list of installed Operators.
3. Click the Update Service tab.
4. From the list of installed OpenShift Update Service applications, select the application to be deleted and then click Delete UpdateService.
5. From the Delete UpdateService? confirmation dialog, click Delete to confirm the deletion.

Procedure

1. Get the OpenShift Update Service application name using the namespace the OpenShift Update Service application was created in, for example, openshift-update-service:

   $ oc get updateservice -n openshift-update-service

   Example output

   NAME      AGE
   service   6s

2. Delete the OpenShift Update Service application using the NAME value from the previous step and the namespace the OpenShift Update Service application was created in, for example, openshift-update-service:

   $ oc delete updateservice service -n openshift-update-service

   Example output

   updateservice.updateservice.operator.openshift.io "service" deleted

Procedure

1. In the web console, click Operators → Installed Operators.
2. Select OpenShift Update Service from the list of installed Operators and click Uninstall Operator.
3. From the Uninstall Operator? confirmation dialog, click Uninstall to confirm the uninstallation.

Procedure

1. Change to the project containing the OpenShift Update Service Operator, for example, openshift-update-service:

   $ oc project openshift-update-service

   Example output

   Now using project "openshift-update-service" on server "https://example.com:6443".

2. Get the name of the OpenShift Update Service Operator operator group:

   $ oc get operatorgroup

   Example output

   NAME                             AGE
   openshift-update-service-fprx2   4m41s

3. Delete the operator group, for example, openshift-update-service-fprx2:

   $ oc delete operatorgroup openshift-update-service-fprx2

   Example output

   operatorgroup.operators.coreos.com "openshift-update-service-fprx2" deleted

4. Get the name of the OpenShift Update Service Operator subscription:

   $ oc get subscription

   Example output

   NAME                      PACKAGE                   SOURCE                        CHANNEL
   update-service-operator   update-service-operator   updateservice-index-catalog   v1

5. Using the Name value from the previous step, check the current version of the subscribed OpenShift Update Service Operator in the currentCSV field:

   $ oc get subscription update-service-operator -o yaml | grep " currentCSV"

   Example output

     currentCSV: update-service-operator.v0.0.1

6. Delete the subscription, for example, update-service-operator:

   $ oc delete subscription update-service-operator

   Example output

   subscription.operators.coreos.com "update-service-operator" deleted

7. Delete the CSV for the OpenShift Update Service Operator using the currentCSV value from the previous step:

   $ oc delete clusterserviceversion update-service-operator.v0.0.1

   Example output

   clusterserviceversion.operators.coreos.com "update-service-operator.v0.0.1" deleted