Red Hat Training

A Red Hat training course is available for OpenShift Container Platform

Chapter 5. Upgrading a Cluster

5.1. Overview

When new versions of OpenShift Container Platform are released, you can upgrade your existing cluster to apply the latest enhancements and bug fixes. This includes upgrading from previous minor versions, such as release 3.2 to 3.3, and applying asynchronous errata updates within a minor version (3.3.z releases). See the OpenShift Container Platform 3.3 Release Notes to review the latest changes.

Note

Due to the core architectural changes between the major versions, OpenShift Enterprise 2 environments cannot be upgraded to OpenShift Container Platform 3 and require a fresh installation.

Unless noted otherwise, node and masters within a major version are forward and backward compatible across one minor version, so upgrading your cluster should go smoothly. However, you should not run mismatched versions longer than necessary to upgrade the entire cluster.

5.1.1. In-place or Blue-Green Upgrades

There are two methods for performing OpenShift Container Platform cluster upgrades. You can either do in-place upgrades (automated or manual), or upgrade using a blue-green deployment method.

In-place Upgrades

With in-place upgrades, the cluster upgrade is performed on all hosts in a single cluster. If you installed using the quick or advanced installation and the ~/.config/openshift/installer.cfg.yml or inventory file that was used is available, you can perform an automated in-place upgrade. Alternatively, you can upgrade in-place manually.

Blue-green Deployments

With blue-green deployments, you can reduce downtime caused while upgrading an environment by creating a parallel environment on which the new deployment can be installed. If a problem is detected, and after the new deployment is verified, traffic can be switched over with the option to rollback.

5.2. Performing Automated In-place Cluster Upgrades

5.2.1. Overview

If you installed using the advanced installation and the inventory file that was used is available, you can use the upgrade playbook to automate the OpenShift cluster upgrade process. If you installed using the quick installation method and a ~/.config/openshift/installer.cfg.yml file is available, you can use the installer to perform the automated upgrade.

The automated upgrade performs the following steps for you:

  • Applies the latest configuration.
  • Upgrades and restart master services.
  • Upgrades and restart node services.
  • Applies the latest cluster policies.
  • Updates the default router if one exists.
  • Updates the default registry if one exists.
  • Updates default image streams and InstantApp templates.
Important

Ensure that you have met all prerequisites before proceeding with an upgrade. Failure to do so can result in a failed upgrade.

Important

Running Ansible playbooks with the --tags or --check options is not supported by Red Hat.

5.2.2. Preparing for an Automated Upgrade

Important

Before upgrading your cluster to OpenShift Container Platform 3.3, the cluster must be already upgraded to the latest asynchronous release of version 3.2. Cluster upgrades cannot span more than one minor version at a time, so if your cluster is at version 3.0 or 3.1, you must first upgrade incrementally (e.g., 3.0 to 3.1, then 3.1 or 3.2).

To prepare for an automated upgrade:

  1. If you are upgrading from version 3.2 to 3.3, manually disable the 3.2 channel and enable the 3.3 channel on each master and node host: 

    # subscription-manager repos --disable="rhel-7-server-ose-3.2-rpms" \
    --enable="rhel-7-server-ose-3.3-rpms"\
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-extras-rpms"
# yum clean all

  2. For any upgrade path, always ensure that you have the latest version of the atomic-openshift-utils package, which should also update the openshift-ansible-* packages: 

    # yum update atomic-openshift-utils

  3. Install or update to the following latest available *-excluder packages on each RHEL 7 system, which helps ensure your systems stay on the correct versions of atomic-openshift and docker packages when you are not trying to upgrade, according to the OpenShift Container Platform version: 

    # yum install atomic-openshift-excluder atomic-openshift-docker-excluder

    These packages add entries to the exclude directive in the host’s /etc/yum.conf file.

  4. You must be logged in as a cluster administrative user on the master host for the upgrade to succeed: 

    $ oc login

After satisfying these steps, there are two methods for running the automated upgrade:

Choose and follow one of these methods.

5.2.3. Using the Installer to Upgrade

If you installed OpenShift Container Platform using the quick installation method, you should have an installation configuration file located at ~/.config/openshift/installer.cfg.yml. The installer requires this file to start an upgrade.

The installer supports upgrading between minor versions of OpenShift Container Platform (one minor version at a time, e.g., 3.2 to 3.3) as well as between asynchronous errata updates within a minor version (e.g., 3.3.z).

If you have an older format installation configuration file in ~/.config/openshift/installer.cfg.yml from an installation of a previous cluster version, the installer will attempt to upgrade the file to the new supported format. If you do not have an installation configuration file of any format, you can create one manually.

To start an upgrade with the quick installer:

  1. Satisfy the steps in Preparing for an Automated Upgrade to ensure you are using the latest upgrade playbooks.

  2. Run the installer with the upgrade subcommand: 

    # atomic-openshift-installer upgrade
  3. Then, follow the on-screen instructions to upgrade to the latest release.
  4. After all master and node upgrades have completed, a recommendation will be printed to reboot all hosts.

  5. After rebooting, if there are no additional features enabled, you can verify the upgrade. Otherwise, the next step depends on what additional features have you previously enabled.

    FeatureNext Step

    Aggregated Logging

    Upgrade the EFK logging stack.

    Cluster Metrics

    Upgrade cluster metrics.

5.2.4. Running the Upgrade Playbook Directly

You can run the automated upgrade playbook using Ansible directly, similar to the advanced installation method, if you have an inventory file.

The same v3_3 upgrade playbook can be used to upgrade either of the following to the latest 3.3 release:

5.2.4.1. Upgrading to OpenShift Container Platform 3.3

To run an upgrade from OpenShift Container Platform 3.2 to 3.3:

  1. Satisfy the steps in Preparing for an Automated Upgrade to ensure you are using the latest upgrade playbooks.
  2. Ensure the deployment_type parameter in your inventory file is set to openshift-enterprise.
  3. If you have multiple masters configured and want to enable rolling, full system restarts of the hosts, you can set the openshift_rolling_restart_mode parameter in your inventory file to system. Otherwise, the default value services performs rolling service restarts on HA masters, but does not reboot the systems. See Configuring Cluster Variables for details.

  4. Run the v3_3 upgrade playbook. If your inventory file is located somewhere other than the default /etc/ansible/hosts, add the -i flag to specify the location. If you previously used the atomic-openshift-installer command to run your installation, you can check ~/.config/openshift/hosts (previously located at ~/.config/openshift/.ansible/hosts) for the last inventory file that was used, if needed. 

    # ansible-playbook [-i </path/to/inventory/file>] \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_3/upgrade.yml
  5. After all master and node upgrades have completed, a recommendation will be printed to reboot all hosts.

  6. After rebooting, if there are no additional features enabled, you can verify the upgrade. Otherwise, the next step depends on what additional features have you previously enabled.

    FeatureNext Step

    Aggregated Logging

    Upgrade the EFK logging stack.

    Cluster Metrics

    Upgrade cluster metrics.

5.2.4.2. Upgrading to OpenShift Container Platform 3.3 Asynchronous Releases

To apply asynchronous errata updates to an existing OpenShift Container Platform 3.3 cluster:

  1. Satisfy the steps in Preparing for an Automated Upgrade to ensure you are using the latest upgrade playbooks.

  2. Run the v3_3 upgrade playbook (the same playbook that is used for upgrading from OpenShift Container Platform 3.2 to 3.3). If your inventory file is located somewhere other than the default /etc/ansible/hosts, add the -i flag to specify the location. If you previously used the atomic-openshift-installer command to run your installation, you can check ~/.config/openshift/hosts (previously located at ~/.config/openshift/.ansible/hosts) for the last inventory file that was used, if needed. 

    # ansible-playbook [-i </path/to/inventory/file>] \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_3/upgrade.yml
  3. After all master and node upgrades have completed, a recommendation will be printed to reboot all hosts.

  4. After rebooting, if there are no additional features enabled, you can verify the upgrade. Otherwise, the next step depends on what additional features have you previously enabled.

    FeatureNext Step

    Aggregated Logging

    Upgrade the EFK logging stack.

    Cluster Metrics

    Upgrade cluster metrics.

5.2.5. Upgrading the EFK Logging Stack

If you have previously deployed the EFK logging stack and want to upgrade to the latest logging component images, the steps must be performed manually as shown in Manual Upgrades.

5.2.6. Upgrading Cluster Metrics

If you have previously deployed cluster metrics, you must manually update to the latest metric components.

5.2.7. Verifying the Upgrade

To verify the upgrade:

  1. First check that all nodes are marked as Ready: 

    # oc get nodes
NAME                        STATUS                     AGE
master.example.com          Ready,SchedulingDisabled   165d
node1.example.com           Ready                      165d
node2.example.com           Ready                      165d

  2. Then, verify that you are running the expected versions of the docker-registry and router images, if deployed. Replace <tag> with v3.3.1.25 for the latest version. 

    # oc get -n default dc/docker-registry -o json | grep \"image\"
    "image": "openshift3/ose-docker-registry:<tag>",
# oc get -n default dc/router -o json | grep \"image\"
    "image": "openshift3/ose-haproxy-router:<tag>",

  3. After upgrading, you can use the diagnostics tool on the master to look for common issues: 

    # oadm diagnostics
...
[Note] Summary of diagnostics execution:
[Note] Completed with no errors or warnings seen.

5.3. Performing Manual In-place Cluster Upgrades

5.3.1. Overview

As an alternative to performing an automated upgrade, you can manually upgrade your OpenShift cluster. To manually upgrade without disruption, it is important to upgrade each component as documented in this topic.

Before you begin your upgrade, familiarize yourself now with the entire procedure. Specific releases may require additional steps to be performed at key points before or during the standard upgrade process.

Important

Ensure that you have met all prerequisites before proceeding with an upgrade. Failure to do so can result in a failed upgrade.

5.3.2. Preparing for a Manual Upgrade

Note

Before upgrading your cluster to OpenShift Container Platform 3.3, the cluster must be already upgraded to the latest asynchronous release of version 3.2. Cluster upgrades cannot span more than one minor version at a time, so if your cluster is at version 3.0 or 3.1, you must first upgrade incrementally (e.g., 3.0 to 3.1, then 3.1 or 3.2).

To prepare for a manual upgrade, follow these steps:

  1. If you are upgrading from version 3.2 to 3.3, manually disable the 3.2 channel and enable the 3.3 channel on each host: 

    # subscription-manager repos --disable="rhel-7-server-ose-3.2-rpms" \
    --enable="rhel-7-server-ose-3.3-rpms" \
    --enable="rhel-7-server-extras-rpms"

    On RHEL 7 systems, also clear the yum cache: 

    # yum clean all

  2. Install or update to the latest available version of the atomic-openshift-utils package on each RHEL 7 system, which provides files that will be used in later sections: 

    # yum install atomic-openshift-utils

  3. Install or update to the following latest available *-excluder packages on each RHEL 7 system, which helps ensure your systems stay on the correct versions of atomic-openshift and docker packages when you are not trying to upgrade, according to the OpenShift Container Platform version: 

    # yum install atomic-openshift-excluder atomic-openshift-docker-excluder

    These packages add entries to the exclude directive in the host’s /etc/yum.conf file.

  4. Create an etcd backup on each master. The etcd package is required, even if using embedded etcd, for access to the etcdctl command to make the backup. The package is installed by default for RHEL Atomic Host 7 systems. If the master is a RHEL 7 system, ensure the package is installed: 

    # yum install etcd

    Then, create the backup: 

    # ETCD_DATA_DIR=/var/lib/origin/openshift.local.etcd 1
# etcdctl backup \
    --data-dir $ETCD_DATA_DIR \
    --backup-dir $ETCD_DATA_DIR.bak.<date> 2
    1
    This directory is for embedded etcd. If you use a separate etcd cluster, use /var/lib/etcd instead.
    2
    Use the date of the backup, or some unique identifier, for <date>. The command will not make a backup if the --backup-dir location already exists.

  5. For any upgrade path, ensure that you are running the latest kernel on each RHEL 7 system: 

    # yum update kernel

5.3.3. Upgrading Master Components

Upgrade your master hosts first:

  1. Run the following command on each master to remove the atomic-openshift packages from the list of yum excludes on the host: 

    # atomic-openshift-excluder unexclude

  2. Upgrade the atomic-openshift packages or related images.

    1. For masters using the RPM-based method on a RHEL 7 system, upgrade all installed atomic-openshift packages: 

      # yum upgrade atomic-openshift\*

    2. For masters using the containerized method on a RHEL 7 or RHEL Atomic Host 7 system, set the IMAGE_VERSION parameter to the version you are upgrading to in the following files:

      • /etc/sysconfig/atomic-openshift-master (single master clusters only)
      • /etc/sysconfig/atomic-openshift-master-controllers (multi-master clusters only)
      • /etc/sysconfig/atomic-openshift-master-api (multi-master clusters only)
      • /etc/sysconfig/atomic-openshift-node

      • /etc/sysconfig/atomic-openshift-openvswitch

        For example: 

        IMAGE_VERSION=<tag>

        Replace <tag> with v3.3 for the latest version.

  3. In OpenShift Container Platform 3.3, protocol buffers are used by default for internal communications between node, masters, and controllers. To configure this, the following stanzas must be altered in the /etc/origin/master-config.yaml file on each master: 

    masterClients:
  externalKubernetesClientConnectionOverrides:
    acceptContentTypes: application/vnd.kubernetes.protobuf,application/json
    contentType: application/vnd.kubernetes.protobuf
    burst: 400
    qps: 200
  externalKubernetesKubeConfig: ""
  openshiftLoopbackClientConnectionOverrides:
    acceptContentTypes: application/vnd.kubernetes.protobuf,application/json
    contentType: application/vnd.kubernetes.protobuf
    burst: 600
    qps: 300
  openshiftLoopbackKubeConfig: openshift-master.kubeconfig

    For more information on protocol buffers, see https://developers.google.com/protocol-buffers/docs/overview.

  4. In OpenShift Container Platform 3.3, the kubernetesMasterConfig.admissionConfig.pluginConfig parameter in the /etc/origin/master-config.yaml file is being deprecated. If you are upgrading from version 3.2 to 3.3 and this parameter is in use, see General Admission Rules for guidance on moving and merging into admissionConfig.pluginConfig.

  5. Restart the master service(s) on each master and review logs to ensure they restart successfully.

    For single master clusters: 

    # systemctl restart atomic-openshift-master
# journalctl -r -u atomic-openshift-master

    For multi-master clusters: 

    # systemctl restart atomic-openshift-master-controllers
# systemctl restart atomic-openshift-master-api
# journalctl -r -u atomic-openshift-master-controllers
# journalctl -r -u atomic-openshift-master-api

  6. Because masters also have node components running on them in order to be configured as part of the OpenShift SDN, restart the atomic-openshift-node and openvswitch services: 

    # systemctl restart atomic-openshift-node
# systemctl restart openvswitch
# journalctl -r -u openvswitch
# journalctl -r -u atomic-openshift-node

  7. Run the following command on each master to add the atomic-openshift packages back to the list of yum excludes on the host: 

    # atomic-openshift-excluder exclude

Upgrade any external etcd hosts using the RPM-based method on a RHEL 7 system:

  1. Upgrade the etcd package: 

    # yum update etcd

  2. Restart the etcd service and review the logs to ensure it restarts successfully: 

    # systemctl restart etcd
# journalctl -r -u etcd
Note

During the cluster upgrade, it can sometimes be useful to take a master out of rotation since some DNS client libraries will not properly to the other masters for cluster DNS. In addition to stopping the master and controller services, you can remove the EndPoint from the Kubernetes service’s subsets.addresses. 

$ oc edit ep/kubernetes -n default

When the master is restarted, the Kubernetes service will be automatically updated.

5.3.4. Updating Policy Definitions

After a cluster upgrade, the recommended default cluster roles may be updated. To check if an update is recommended for your environment, you can run: 

# oadm policy reconcile-cluster-roles
Warning

If you have customized default cluster roles and want to ensure a role reconciliation does not modify those customized roles, annotate them with openshift.io/reconcile-protect set to true. Doing so means you are responsible for manually updating those roles with any new or required permissions during upgrades.

This command outputs a list of roles that are out of date and their new proposed values. For example: 

# oadm policy reconcile-cluster-roles
apiVersion: v1
items:
- apiVersion: v1
  kind: ClusterRole
  metadata:
    creationTimestamp: null
    name: admin
  rules:
  - attributeRestrictions: null
    resources:
    - builds/custom
...
Note

Your output will vary based on the OpenShift version and any local customizations you have made. Review the proposed policy carefully.

You can either modify this output to re-apply any local policy changes you have made, or you can automatically apply the new policy using the following process:

  1. Reconcile the cluster roles: 

    # oadm policy reconcile-cluster-roles \
    --additive-only=true \
    --confirm

  2. Reconcile the cluster role bindings: 

    # oadm policy reconcile-cluster-role-bindings \
    --exclude-groups=system:authenticated \
    --exclude-groups=system:authenticated:oauth \
    --exclude-groups=system:unauthenticated \
    --exclude-users=system:anonymous \
    --additive-only=true \
    --confirm

  3. Reconcile security context constraints: 

    # oadm policy reconcile-sccs \
    --additive-only=true \
    --confirm

5.3.5. Upgrading Nodes

After upgrading your masters, you can upgrade your nodes. When restarting the atomic-openshift-node service, there will be a brief disruption of outbound network connectivity from running pods to services while the service proxy is restarted. The length of this disruption should be very short and scales based on the number of services in the entire cluster.

Note

Blue-green deployments are another proven approach to reducing downtime caused while updating an environment.

One at at time for each node that is not also a master, you must disable scheduling and evacuate its pods to other nodes, then upgrade packages and restart services.

  1. Run the following command on each node to remove the atomic-openshift packages from the list of yum excludes on the host: 

    # atomic-openshift-excluder unexclude

  2. As a user with cluster-admin privileges, disable scheduling for the node: 

    # oadm manage-node <node> --schedulable=false

  3. Evacuate pods on the node to other nodes:

    Important

    The --force option deletes any pods that are not backed by a replication controller. 

    # oadm manage-node <node> --evacuate --force

  4. Upgrade the node component packages or related images.

    1. For nodes using the RPM-based method on a RHEL 7 system, upgrade all installed atomic-openshift packages: 

      # yum upgrade atomic-openshift\*

    2. For nodes using the containerized method on a RHEL 7 or RHEL Atomic Host 7 system, set the IMAGE_VERSION parameter in the /etc/sysconfig/atomic-openshift-node and /etc/sysconfig/openvswitch files to the version you are upgrading to. For example: 

      IMAGE_VERSION=<tag>

      Replace <tag> with v3.3 for the latest version.

  5. In OpenShift Container Platform 3.3, protocol buffers are used by default for internal communications between node, masters, and controllers. To configure this, the following stanzas must be altered in the /etc/origin/node-config.yaml file on each node: 

    masterClientConnectionOverrides:
  acceptContentTypes: application/vnd.kubernetes.protobuf,application/json
  contentType: application/vnd.kubernetes.protobuf
  burst: 200
  qps: 100

    For more information on protocol buffers, see https://developers.google.com/protocol-buffers/docs/overview.

  6. Restart the atomic-openshift-node and openvswitch services and review the logs to ensure they restart successfully: 

    # systemctl restart atomic-openshift-node
# systemctl restart openvswitch
# journalctl -r -u atomic-openshift-node
# journalctl -r -u openvswitch

  7. Re-enable scheduling for the node: 

    # oadm manage-node <node> --schedulable

  8. Run the following command on the node to add the atomic-openshift packages back to the list of yum excludes on the host: 

    # atomic-openshift-excluder exclude
  9. Repeat these steps on the next node, and continue repeating these steps until all nodes have been upgraded.

  10. After all nodes have been upgraded, as a user with cluster-admin privileges, verify that all nodes are showing as Ready: 

    # oc get nodes
NAME                        STATUS                     AGE
master.example.com          Ready,SchedulingDisabled   165d
node1.example.com           Ready                      165d
node2.example.com           Ready                      165d

5.3.6. Upgrading the Router

If you have previously deployed a router, the router deployment configuration must be upgraded to apply updates contained in the router image. To upgrade your router without disrupting services, you must have previously deployed a highly-available routing service.

Note

If you previously customized your HAProxy routing template, then, depending on the changes, additional steps may be required due to changes in the routing data structure starting in OpenShift Container Platform 3.3. See Routing Data Structure Changes in the OpenShift Container Platform 3.3 Release Notes for details.

Edit your router’s deployment configuration. For example, if it has the default router name: 

# oc edit dc/router

Apply the following changes: 

...
spec:
 template:
    spec:
      containers:
      - env:
        ...
        image: registry.access.redhat.com/openshift3/ose-haproxy-router:<tag> 1
        imagePullPolicy: IfNotPresent
        ...
1
Adjust <tag> to match the version you are upgrading to (use v3.3 for the latest version).

You should see one router pod updated and then the next.

5.3.7. Upgrading the Registry

The registry must also be upgraded for changes to take effect in the registry image. If you have used a PersistentVolumeClaim or a host mount point, you may restart the registry without losing the contents of your registry. Storage for the Registry details how to configure persistent storage for the registry.

Edit your registry’s deployment configuration: 

# oc edit dc/docker-registry

Apply the following changes: 

...
spec:
 template:
    spec:
      containers:
      - env:
        ...
        image: registry.access.redhat.com/openshift3/ose-docker-registry:<tag> 1
        imagePullPolicy: IfNotPresent
        ...
1
Adjust <tag> to match the version you are upgrading to (use v3.3 for the latest version).
Important

Images that are being pushed or pulled from the internal registry at the time of upgrade will fail and should be restarted automatically. This will not disrupt pods that are already running.

5.3.7.1. Updating Custom Registry Configuration Files

Note

You may safely skip this part if you do not use a custom registry configuration file.

The internal Docker registry version 3.3.0 and higher requires following entries in the middleware section of the configuration file: 

middleware:
  registry:
    - name: openshift
  repository:
    - name: openshift
  storage:
    - name: openshift
  1. Edit your custom configuration file, adding the missing entries.
  2. Deploy your updated configuration.
  3. Append the --overwrite flag to oc volume dc/docker-registry --add to replace a volume mount of your previous secret.
  4. You can safely remove the old secret.

5.3.7.2. Enforcing Quota in the Registry

Quota must be enforced to prevent layer blobs that exceed the size limit from being written to the registry’s storage. This can be achieved via a configuration file: 

...
middleware:
  repository:
    - name: openshift
      options:
        enforcequota: true
...

Alternatively, use the REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA environment variable, which is set to true for the new registry deployments by default. Existing deployments need to be modified using: 

# oc set env dc/docker-registry REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA=true

5.3.8. Updating the Default Image Streams and Templates

By default, the quick and advanced installation methods automatically create default image streams, InstantApp templates, and database service templates in the openshift project, which is a default project to which all users have view access. These objects were created during installation from the JSON files located under the /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/ directory.

Note

Because RHEL Atomic Host 7 cannot use yum to update packages, the following steps must take place on a RHEL 7 system.

  1. Update the packages that provide the example JSON files. On a subscribed Red Hat Enterprise Linux 7 system where you can run the CLI as a user with cluster-admin permissions, install or update to the latest version of the atomic-openshift-utils package, which should also update the openshift-ansible- packages: 

    # yum update atomic-openshift-utils

    The openshift-ansible-roles package provides the latest example JSON files.

  2. After a manual upgrade, get the latest templates from openshift-ansible-roles: 

    rpm -ql openshift-ansible-roles | grep examples | grep v1.3

    In this example, /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/image-streams/image-streams-rhel7.json is the latest file that you want in the latest openshift-ansible-roles package.

    /usr/share/openshift/examples/image-streams/image-streams-rhel7.json is not owned by a package, but is updated by Ansible. If you are upgrading outside of Ansible. you need to get the latest .json files on the system where you are running oc, which can run anywhere that has access to the master.

  3. Install atomic-openshift-utils and its dependencies to install the new content into /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/.: 

    $ oc create -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/image-streams/image-streams-rhel7.json
$ oc create -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/image-streams/dotnet_imagestreams.json
$ oc replace -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/image-streams/image-streams-rhel7.json
$ oc replace -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/image-streams/dotnet_imagestreams.json

  4. Update the templates: 

    $ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/quickstart-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/db-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/infrastructure-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/xpaas-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/xpaas-streams/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/quickstart-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/db-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/infrastructure-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/xpaas-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/xpaas-streams/

    Errors are generated for items that already exist. This is expected behavior: 

    # oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/quickstart-templates/
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/quickstart-templates/cakephp-mysql.json": templates "cakephp-mysql-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/quickstart-templates/cakephp.json": templates "cakephp-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/quickstart-templates/dancer-mysql.json": templates "dancer-mysql-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/quickstart-templates/dancer.json": templates "dancer-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.3/quickstart-templates/django-postgresql.json": templates "django-psql-example" already exists

Now, content can be updated. Without running the automated upgrade playbooks, the content is not updated in /usr/share/openshift/.

5.3.9. Importing the Latest Images

After updating the default image streams, you may also want to ensure that the images within those streams are updated. For each image stream in the default openshift project, you can run: 

# oc import-image -n openshift <imagestream>

For example, get the list of all image streams in the default openshift project: 

# oc get is -n openshift
NAME     DOCKER REPO                                                      TAGS                   UPDATED
mongodb  registry.access.redhat.com/openshift3/mongodb-24-rhel7           2.4,latest,v3.1.1.6    16 hours ago
mysql    registry.access.redhat.com/openshift3/mysql-55-rhel7             5.5,latest,v3.1.1.6    16 hours ago
nodejs   registry.access.redhat.com/openshift3/nodejs-010-rhel7           0.10,latest,v3.1.1.6   16 hours ago
...

Update each image stream one at a time: 

# oc import-image -n openshift nodejs
The import completed successfully.

Name:			nodejs
Created:		10 seconds ago
Labels:			<none>
Annotations:		openshift.io/image.dockerRepositoryCheck=2016-07-05T19:20:30Z
Docker Pull Spec:	172.30.204.22:5000/openshift/nodejs

Tag	Spec								Created		PullSpec						Image
latest	4								9 seconds ago	registry.access.redhat.com/rhscl/nodejs-4-rhel7:latest	570ad8ed927fd5c2c9554ef4d9534cef808dfa05df31ec491c0969c3bd372b05
4	registry.access.redhat.com/rhscl/nodejs-4-rhel7:latest		9 seconds ago	<same>							570ad8ed927fd5c2c9554ef4d9534cef808dfa05df31ec491c0969c3bd372b05
0.10	registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest	9 seconds ago	<same>							a1ef33be788a28ec2bdd48a9a5d174ebcfbe11c8e986d2996b77f5bccaaa4774
Important

In order to update your S2I-based applications, you must manually trigger a new build of those applications after importing the new images using oc start-build <app-name>.

5.3.10. Upgrading the EFK Logging Stack

Use the following to upgrade an already-deployed EFK logging stack.

Note

The following steps apply when upgrading to OpenShift Container Platform 3.3+.

  1. Ensure you are working in the project where the EFK stack was previously deployed. For example, if the project is named logging: 

    $ oc project logging

  2. Recreate the deployer templates for service accounts and running the deployer: 

    $ oc apply -n openshift -f \
    usr/share/ansible/openshift-ansible/roles/openshift_hosted_templates/files/v1.3/enterprise/metrics-deployer.yaml

  3. Generate any missing service accounts and roles: 

    $ oc process logging-deployer-account-template | oc apply -f -

  4. Ensure that the cluster role oauth-editor is assigned to the logging-deployer service account: 

    $ oadm policy add-cluster-role-to-user oauth-editor \
       system:serviceaccount:logging:logging-deployer

  5. In preparation for running the deployer, ensure that you have the configurations for your current deployment in the logging-deployer ConfigMap.

    Important

    Ensure that your image version is the latest version, not the currently installed version.

  6. Run the deployer with the parameter in upgrade mode: 

    $ oc new-app logging-deployer-template -p MODE=upgrade

    Running the deployer in this mode handles scaling down the components to minimize loss of logs, patching configurations, generating missing secrets and keys, and scaling the components back up to their previous replica count.

    Important

    Due to the privileges needed to label and unlabel a node for controlling the deployment of Fluentd pods, the deployer does delete the logging-fluentd Daemonset and recreates it from the logging-fluentd-template template.

5.3.11. Upgrading Cluster Metrics

After upgrading an already-deployed Cluster Metrics install, you must update to a newer version of the metrics components.

  • The update process stops all the metrics containers, updates the metrics configuration files, and redeploys the newer components.
  • It does not change the metrics route.
  • It does not delete the metrics persistent volume claim. Metrics stored to persistent volumes before the update are available after the update completes.
Important

The update deletes all non-persisted metric values and overwrites local changes to the metrics configurations. For example, the number of instances in a replica set is not saved.

To update, follow the same steps as when the metrics components were first deployed, using the correct template, except this time, specify the MODE=refresh option: 

$ oc new-app -f metrics-deployer.yaml \
    -p HAWKULAR_METRICS_HOSTNAME=hm.example.com,MODE=refresh 1
1
In the original deployment command, there was no MODE=refresh.
Note

During the update, the metrics components do not run. Because of this, they cannot collect data and a gap normally appears in the graphs.

5.3.12. Additional Manual Steps Per Release

Some OpenShift Container Platform releases may have additional instructions specific to that release that must be performed to fully apply the updates across the cluster. This section will be updated over time as new asynchronous updates are released for OpenShift Container Platform 3.3.

As of the latest 3.3 release (v3.3), there are no 3.3 asynchronous releases that require additional instructions during upgrade. See the OpenShift Container Platform 3.3 Release Notes to review the latest release notes.

5.3.13. Verifying the Upgrade

To verify the upgrade, first check that all nodes are marked as Ready: 

# oc get nodes
NAME                        STATUS                     AGE
master.example.com          Ready,SchedulingDisabled   165d
node1.example.com           Ready                      165d
node2.example.com           Ready                      165d

Then, verify that you are running the expected versions of the docker-registry and router images, if deployed. Replace <tag> with v3.3 for the latest version. 

# oc get -n default dc/docker-registry -o json | grep \"image\"
    "image": "openshift3/ose-docker-registry:<tag>",
# oc get -n default dc/router -o json | grep \"image\"
    "image": "openshift3/ose-haproxy-router:<tag>",

After upgrading, you can use the diagnostics tool on the master to look for common issues: 

# oadm diagnostics
...
[Note] Summary of diagnostics execution:
[Note] Completed with no errors or warnings seen.

5.4. Blue-Green Deployments

5.4.1. Overview

Note

This topic serves as an alternative node upgrade method to the approach in Manual In-place Upgrades.

Blue-green deployments are a proven approach to reducing downtime caused while upgrading an environment. This is done by creating a parallel environment on which the new deployment can be installed. If a problem is detected, and after the new deployment is verified, traffic can be switched over with the option to rollback.

While blue-green is a valid strategy for deploying just about any software, there are always trade-offs. Not all environments have the same uptime requirements or the resources to properly perform blue-green deployments. In an OpenShift Container Platform environment, the most suitable candidate for blue-green deployments are the nodes. All user processes run on these systems and even critical pieces of OpenShift Container Platform infrastructure are self-hosted there. Uptime is most important for these workloads and the additional complexity of blue-green deployments can be justified. The exact implementation of this approach varies based on your requirements. Often the main challenge is having the excess capacity to facilitate such an approach.

Another lesser challenge is that the administrator must temporarily share the Red Hat software entitlements between the blue-green deployments or provide access to the installation content by means of a system such as Red Hat Satellite. This can be accomplished by sharing the consumer ID from the previous host.

5.4.2. Preparing for Upgrade

  1. On the old host: 

    # subscription-manager identity | grep system
system identity: 6699375b-06db-48c4-941e-689efd6ce3aa

  2. On the new host: 

    # subscription-manager register --consumerid=6699375b-06db-48c4-941e-689efd6ce3aa
    Important

    After a successful deployment, remember to unregister the old host with subscription-manager clean to prevent the environment from being out of compliance.

  3. After the master and etcd servers have been upgraded, you must ensure that your current production nodes are labeled either blue or green. In this example, the current installation will be blue and the new environment will be green. On each production node in your current installation: 

    $ oc label --all nodes color=blue

    In the case of nodes requiring the uptime guarantees of a blue-green deployment, the -l flag can be used to match a subset of the environment using a selector.

  4. Create the new green environment for any nodes that are to be replaced by adding an equal number of new nodes to the existing cluster. Ansible can apply the color=green label using the openshift_node_labels variable for each node.
  5. In order to delay workload scheduling until the nodes are healthy, be sure to set the openshift_schedulable=false variable. After the green nodes are in Ready state, they can be made schedulable.

  6. Blue nodes are disabled so that no new pods are run on them: 

    # oadm manage-node --schedulable=true --selector=color=green
# oadm manage-node --schedulable=false --selector=color=blue

A common practice is to scale the registry and router pods until they are migrated to the green nodes. For these pods, a canary deployment approach is commonly used. Scaling them up will make them immediately active on the new nodes. Pointing the deployment configuration to the new image initiates a rolling update. However, because of node anti-affinity, and the fact that the blue nodes are still unschedulable, the deployments to the old nodes will fail. At this point, the registry and router deployments can be scaled down to the original number of pods. At any given point, the original number of pods is still available so no capacity is lost.

5.4.3. Warming the New Nodes

In order for pods to be migrated from the blue environment to the green, the images must be pulled. Network latency and load on the registry can cause delays if there is not sufficient capacity built in to the environment. Often, the best way to minimize impact to the running system is to trigger new pod deployments that will land on the new nodes. Accomplish this by importing new image streams.

A major release of OpenShift Container Platform is the motivation for a blue-green deployment. At that time, new image streams become available for users of Source-to-Image (S2I). Upon import, any builds or deployments configured with ImageChangeTriggers are automatically created.

Note

It is important to realize that this process can trigger a large number of builds. The good news is that the builds are performed on the green nodes and, therefore, do not impact any traffic on the blue deployment.

To monitor build progress across all namespaces (projects) in the cluster: 

$ oc get events -w --all-namespaces

In large environments, builds rarely completely stop. However, you should see a large increase and decrease caused by the administrative import.

Another benefit of triggering the builds is that it does a fairly good job of fetching the majority of the ancillary images to all nodes such as the various build images, the pod infrastructure image, and deployers. Everything else can be moved over using node evacuation and will proceed more quickly as a result.

5.4.4. Node Evacuation

For larger deployments, it is possible to have other labels that help determine how evacuation can be coordinated. The most conservative approach for avoiding downtime is to evacuate one node at a time. If services are composed of pods using zone anti-affinity, then an entire zone can be evacuated at once. It is important to ensure that the storage volumes used are available in the new zone as this detail can vary among cloud providers.

In OpenShift Enterprise 3.2 and later, a node evacuation is triggered whenever the service is stopped. Achieve manual evacuation and deletion of all blue nodes at once by: 

# oadm manage-node --selector=color=blue --evacuate
# oc delete node --selector=color=blue

5.5. Operating System Updates and Upgrades

5.5.1. Updating and Upgrading the Operating System

Updating or upgrading your operating system (OS), by either changing OS versions or updating the system software, can impact the OpenShift Container Platform software running on those machines. In particular, these updates can affect the iptables rules or ovs flows that OpenShift Container Platform requires to operate.

Use the following to safely upgrade the OS on a host:

  1. Ensure the host is unschedulable, meaning that no new pods will be placed onto the host: 

    $ oadm manage-node <node_name> --schedulable=false

  2. Migrate the pods from the host: 

    $ oadm drain <node_name> --force --delete-local-data --ignore-daemonsets

  3. Install or update the *-excluder packages on each host with the following. This ensures the hosts stay on the correct versions of OpenShift Container Platform, as per the atomic-openshift and docker packages, instead of the most current versions: 

    # yum install atomic-openshift-excluder atomic-openshift-docker-excluder

    This adds entries to the exclude directive in the host’s /etc/yum.conf file.

  4. Update or upgrade the host packages, and reboot the host. A reboot ensures that the host is running the newest versions, and means that the docker and OpenShift Container Platform processes have been restarted, which will force them to check that all of the rules in other services are correct.

    However, instead of rebooting a node host, you can restart the services that are affected, or preserve the iptables state. Both processes are described in the OpenShift Container Platform IPtables topic. The ovs flow rules do not need to be saved, but restarting the OpenShift Container Platform node software will fix the flow rules.

  5. Configure the host to be schedulable again: 

    $ oadm manage-node <node_name> --schedulable=true