Chapter 2. Performing Automated In-place Cluster Upgrades

If you installed using the advanced installation and the inventory file that was used is available, you can use upgrade playbooks to automate the OpenShift cluster upgrade process.

The OpenShift Container Platform 3.9 release includes a merge of features and fixes from Kubernetes 1.8 and 1.9. As a result, the upgrade process from OpenShift Container Platform 3.7 completes with the cluster fully upgraded to OpenShift Container Platform 3.9, seemingly "skipping" the 3.8 release. Technically, the OpenShift Container Platform 3.7 cluster is first upgraded to 3.8-versioned packages, and then the process immediately continues upgrading to OpenShift Container Platform 3.9 automatically. Your cluster should only remain at 3.8-versioned packages for as long as it takes to successfully complete the upgrade to OpenShift Container Platform 3.9.

The automated 3.7 to 3.9 control plane upgrade performs the following steps for you:

The automated 3.7 to 3.9 node upgrade performs a rolling update of nodes, which:

Automated upgrade playbooks are run via Ansible directly using the ansible-playbook command with an inventory file, similar to the advanced installation method. The same v3_9 upgrade playbooks can be used for either of the following scenarios:

If you are completing a large-scale upgrade, which involves at least 10 worker nodes and thousands of projects and pods, review Special considerations for large-scale upgrades to prevent upgrade failures.

To prepare for an automated upgrade:

After satisfying these steps, you can review the following sections for more information on how the upgrade process works and make decisions on additional upgrade customization options if you so choose. When you are prepared to run the upgrade, you can continue to Upgrading to the Latest OpenShift Container Platform 3.9 Release.

$ ansible-playbook -i <path/to/inventory/file> \
    </path/to/upgrade/playbook> \
    -e openshift_upgrade_nodes_serial="20%"

$ ansible-playbook -i <path/to/inventory/file> \
    </path/to/upgrade/playbook> \
    -e openshift_upgrade_nodes_serial="2" \
    -e openshift_upgrade_nodes_label="region=group1"

$ ansible-playbook -i <path/to/inventory/file> \
    </path/to/upgrade/playbook> \
    -e openshift_upgrade_nodes_serial=10 \
    -e openshift_upgrade_nodes_max_fail_percentage=20 \
    -e openshift_upgrade_nodes_drain_timeout=600

[OSEv3:vars]
openshift_master_upgrade_pre_hook=/usr/share/custom/pre_master.yml
openshift_master_upgrade_hook=/usr/share/custom/master.yml
openshift_master_upgrade_post_hook=/usr/share/custom/post_master.yml

---
# Trivial example forcing an operator to ack the start of an upgrade
# file=/usr/share/custom/pre_master.yml

- name: note the start of a master upgrade
  debug:
      msg: "Master upgrade of {{ inventory_hostname }} is about to start"

- name: require an operator agree to start an upgrade
  pause:
      prompt: "Hit enter to start the master upgrade"

To upgrade an existing OpenShift Container Platform 3.7 or 3.9 cluster to the latest 3.9 release:

To upgrade an existing EFK logging stack deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/openshift-logging/config.yml Ansible playbook. This is the playbook to use if you were deploying logging for the first time on an existing cluster, but is also used to upgrade existing logging deployments.

        image: <image_name>:<vX.Y>
        imagePullPolicy: IfNotPresent

To upgrade an existing cluster metrics deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/openshift-metrics/config.yml Ansible playbook. This is the playbook to use if you were deploying metrics for the first time on an existing cluster, but is also used to upgrade existing metrics deployments.

For large-scale cluster upgrades, which involve at least 10 worker nodes and thousands of projects and pods, the API object storage migration should be performed prior to running the upgrade playbooks, and then again after the upgrade has successfully completed. Otherwise, the upgrade process will fail.

Refer to the Running the pre- and post- API server model object migration outside of the upgrade window section of the Recommendations for large-scale OpenShift upgrades for further guidance.

Before you upgrade a mixed environment, such as one with Red Hat Enterprise Linux (RHEL) and RHEL Atomic Host, set values in the inventory file for both the openshift_pkg_version and openshift_image_tag parameters. Setting these values ensures that all nodes in your cluster run the same version of OpenShift Container Platform.

For example, to upgrade from OpenShift Container Platform 3.7 to OpenShift Container Platform 3.9, set the following parameters and values:

openshift_pkg_version=-3.9.74
openshift_image_tag=v3.9.74

When upgrading OpenShift Container Platform, you must upgrade the set of nodes where GlusterFS pods are running.

Use care when upgrading these nodes, as drain and unschedule will not terminate and evacuate the GlusterFS pods because they are running as part of a daemonset.

There is also the potential for someone to run an upgrade on multiple nodes at the same time. If this occurs, it would lead to data availability issues if more than one node was hosting GlusterFS pods.

Even if a serial upgrade is running, there is no guarantee sufficient time will be given for GlusterFS to complete all of its healing operations before GlusterFS on the next node is terminated. This could leave the cluster in a bad or unknown state. Therefore, the following procedure is recommended.

Because the default gcePD storage provider uses an RWO (Read-Write Only) access mode, you cannot perform a rolling upgrade on the registry or scale the registry to multiple pods. Therefore, when upgrading OpenShift Container Platform, you must specify the following environment variables in your Ansible inventory file:

[OSEv3:vars]

openshift_hosted_registry_storage_provider=gcs
openshift_hosted_registry_storage_gcs_bucket=bucket01
openshift_hosted_registry_storage_gcs_keyfile=test.key
openshift_hosted_registry_storage_gcs_rootdirectory=/registry

Ensure that the:

To verify the upgrade: