Chapter 3. Upgrading

3.1. Overview

When new versions of OpenShift are released, you can upgrade your existing cluster to apply the latest enhancements and bug fixes. This includes upgrading from previous minor versions, such as release 3.0 to 3.1, and applying asynchronous errata updates within a minor version (3.1.z releases). See the OpenShift Enterprise 3.1 Release Notes to review the latest changes.

Note

Due to the core architectural changes between the major versions, OpenShift Enterprise 2 environments cannot be upgraded to OpenShift Enterprise 3 and require a fresh installation.

Unless noted otherwise, node and masters within a major version are forward and backward compatible, so upgrading your cluster should go smoothly. However, you should not run mismatched versions longer than necessary to upgrade the entire cluster.

If you installed using the quick or advanced installation and the ~/.config/openshift/installer.cfg.yml or inventory file that was used is available, you can perform an automated upgrade. Alternatively, you can upgrade OpenShift manually.

Note

The Upgrading topics pertains to RPM-based installations only (i.e., the quick and advanced installation methods) and does not currently cover container-based installations.

If you are using the Pacemaker method for high availability (HA) masters, you can upgrade to the native HA method either using Ansible playbooks or manually.

3.2. Performing Automated Cluster Upgrades

3.2.1. Overview

Starting with OpenShift 3.0.2, if you installed using the advanced installation and the inventory file that was used is available, you can use the upgrade playbook to automate the OpenShift cluster upgrade process. If you installed using the quick installation method and a ~/.config/openshift/installer.cfg.yml file is available, you can use the installer to perform the automated upgrade.

The automated upgrade performs the following steps for you:

  • Applies the latest configuration.
  • Upgrades and restart master services.
  • Upgrades and restart node services.
  • Applies the latest cluster policies.
  • Updates the default router if one exists.
  • Updates the default registry if one exists.
  • Updates default image streams and InstantApp templates.
Important

Running Ansible playbooks with the --tags or --check options is not supported by Red Hat.

3.2.2. Preparing for an Automated Upgrade

  1. If you are upgrading from OpenShift Enterprise 3.0 to 3.1, on each master and node host you must manually disable the 3.0 channel and enable the 3.1 channel: 

    # subscription-manager repos --disable="rhel-7-server-ose-3.0-rpms" \
    --enable="rhel-7-server-ose-3.1-rpms" \
    --enable="rhel-7-server-rpms"

  2. For any upgrade path, always ensure that you have the latest version of the atomic-openshift-utils package, which should also update the openshift-ansible-* packages: 

    # yum update atomic-openshift-utils

  3. Install or update to the following latest available *-excluder packages on each RHEL 7 system, which helps ensure your systems stay on the correct versions of atomic-openshift and docker packages when you are not trying to upgrade, according to the OpenShift Enterprise version: 

    # yum install atomic-openshift-excluder atomic-openshift-docker-excluder

    These packages add entries to the exclude directive in the host’s /etc/yum.conf file.

  4. You must be logged in as a cluster administrative user on the master host for the upgrade to succeed: 

    $ oc login

There are two methods for running the automated upgrade: using the installer or running the upgrade playbook directly. Choose and follow one method.

3.2.3. Using the Installer to Upgrade

If you installed OpenShift using the quick installation method, you should have an installation configuration file located at ~/.config/openshift/installer.cfg.yml. The installer requires this file to start an upgrade.

Note

The installer currently only supports upgrading from OpenShift Enterprise 3.0 to 3.1. See Upgrading to OpenShift Enterprise 3.1 Asynchronous Releases for instructions on using Ansible directly.

If you have an older format installation configuration file in ~/.config/openshift/installer.cfg.yml from an existing OpenShift Enterprise 3.0 installation, the installer will attempt to upgrade the file to the new supported format. If you do not have an installation configuration file of any format, you can create one manually.

To start the upgrade, run the installer with the upgrade subcommand:

  1. Satisfy the steps in Preparing for an Automated Upgrade to ensure you are using the latest upgrade playbooks.

  2. Run the following command on each host to remove the atomic-openshift packages from the list of yum excludes on the host: 

    # atomic-openshift-excluder unexclude

  3. Run the installer with the upgrade subcommand: 

    # atomic-openshift-installer upgrade
  4. Follow the on-screen instructions to upgrade to the latest release.

  5. After all master and node upgrades have completed, a recommendation will be printed to reboot all hosts. Before rebooting, run the following command on each master and node host to add the atomic-openshift packages back to the list of yum excludes on the host: 

    # atomic-openshift-excluder exclude

    Then reboot all hosts.

  6. After rebooting, continue to Updating Master and Node Certificates.

3.2.4. Running the Upgrade Playbook Directly

Alternatively, you can run the upgrade playbook with Ansible directly, similar to the advanced installation method, if you have an inventory file.

3.2.4.1. Upgrading to OpenShift Enterprise 3.1.0

Before running the upgrade, first update your inventory file to change the deployment_type parameter from enterprise to openshift-enterprise; this is required when upgrading from OpenShift Enterprise 3.0 to 3.1:

Before running the upgrade, first ensure the deployment_type parameter in your inventory file is set to openshift-enterprise.

If you have multiple masters configured and want to enable rolling, full system restarts of the hosts, you can set the openshift_rolling_restart_mode parameter in your inventory file to system. Otherwise, the default value services performs rolling service restarts on HA masters, but does not reboot the systems. See Configuring Cluster Variables for details.

Then, run the v3_0_to_v3_1 upgrade playbook. If your inventory file is located somewhere other than the default /etc/ansible/hosts, add the -i flag to specify the location. If you previously used the atomic-openshift-installer command to run your installation, you can check ~/.config/openshift/.ansible/hosts for the last inventory file that was used, if needed. 

# ansible-playbook [-i </path/to/inventory/file>] \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_0_to_v3_1/upgrade.yml

When the upgrade finishes, a recommendation will be printed to reboot all hosts. After rebooting, continue to Updating Master and Node Certificates.

3.2.4.2. Upgrading to OpenShift Enterprise 3.1 Asynchronous Releases

To apply asynchronous errata updates to an existing OpenShift Enterprise 3.1 cluster, first upgrade the atomic-openshift-utils package on the Red Hat Enterprise Linux 7 system where you will be running Ansible: 

# yum update atomic-openshift-utils

Then, run the v3_1_minor upgrade playbook. If your inventory file is located somewhere other than the default /etc/ansible/hosts, add the -i flag to specify the location. If you previously used the atomic-openshift-installer command to run your installation, you can check ~/.config/openshift/.ansible/hosts for the last inventory file that was used, if needed. 

# ansible-playbook [-i </path/to/inventory/file>] \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_1_minor/upgrade.yml

When the upgrade finishes, a recommendation will be printed to reboot all hosts. After rebooting, continue to Verifying the Upgrade.

3.2.5. Updating Master and Node Certificates

The following steps may be required for any OpenShift cluster that was originally installed prior to the OpenShift Enterprise 3.1 release. This may include any and all updates from that version.

3.2.5.1. Node Certificates

With the 3.1 release, certificates for each of the kubelet nodes were updated to include the IP address of the node. Any node certificates generated before the 3.1 release may not contain the IP address of the node.

If a node is missing the IP address as part of its certificate, clients may refuse to connect to the kubelet endpoint. Usually this will result in errors regarding the certificate not containing an IP SAN.

In order to remedy this situation, you may need to manually update the certificates for your node.

3.2.5.1.1. Checking the Node’s Certificate

The following command can be used to determine which Subject Alternative Names (SANs) are present in the node’s serving certificate. In this example, the Subject Alternative Names are mynode, mynode.mydomain.com, and 1.2.3.4: 

# openssl x509 -in /etc/origin/node/server.crt -text -noout | grep -A 1 "Subject Alternative Name"
X509v3 Subject Alternative Name:
DNS:mynode, DNS:mynode.mydomain.com, IP: 1.2.3.4

Ensure that the nodeIP value set in the /etc/origin/node/node-config.yaml file is present in the IP values from the Subject Alternative Names listed in the node’s serving certificate. If the nodeIP is not present, then it will need to be added to the node’s certificate.

If the nodeIP value is already contained within the Subject Alternative Names, then no further steps are required.

You will need to know the Subject Alternative Names and nodeIP value for the following steps.

3.2.5.1.2. Generating a New Node Certificate

If your current node certificate does not contain the proper IP address, then you must regenerate a new certificate for your node.

Important

Node certificates will be regenerated on the master (or first master) and are then copied into place on node systems.

  1. Create a temporary directory in which to perform the following steps: 

    # mkdir /tmp/node_certificate_update
# cd /tmp/node_certificate_update

  2. Export the signing options: 

    # export signing_opts="--signer-cert=/etc/origin/master/ca.crt \
    --signer-key=/etc/origin/master/ca.key \
    --signer-serial=/etc/origin/master/ca.serial.txt"

  3. Generate the new certificate: 

    # oadm ca create-server-cert --cert=server.crt \
  --key=server.key $signing_opts \
  --hostnames=<existing_SANs>,<nodeIP>

    For example, if the Subject Alternative Names from before were mynode, mynode.mydomain.com, and 1.2.3.4, and the nodeIP was 10.10.10.1, then you would need to run the following command: 

    # oadm ca create-server-cert --cert=server.crt \
  --key=server.key $signing_opts \
  --hostnames=mynode,mynode.mydomain.com,1.2.3.4,10.10.10.1
3.2.5.1.3. Replace Node Serving Certificates

Back up the existing /etc/origin/node/server.crt and /etc/origin/node/server.key files for your node: 

# mv /etc/origin/node/server.crt /etc/origin/node/server.crt.bak
# mv /etc/origin/node/server.key /etc/origin/node/server.key.bak

You must now copy the new server.crt and server.key created in the temporary directory during the previous step: 

# mv /tmp/node_certificate_update/server.crt /etc/origin/node/server.crt
# mv /tmp/node_certificate_update/server.key /etc/origin/node/server.key

After you have replaced the node’s certificate, restart the node service: 

# systemctl restart atomic-openshift-node

3.2.5.2. Master Certificates

With the 3.1 release, certificates for each of the masters were updated to include all names that pods may use to communicate with masters. Any master certificates generated before the 3.1 release may not contain these additional service names.

3.2.5.2.1. Checking the Master’s Certificate

The following command can be used to determine which Subject Alternative Names (SANs) are present in the master’s serving certificate. In this example, the Subject Alternative Names are mymaster, mymaster.mydomain.com, and 1.2.3.4: 

# openssl x509 -in /etc/origin/master/master.server.crt -text -noout | grep -A 1 "Subject Alternative Name"
X509v3 Subject Alternative Name:
DNS:mymaster, DNS:mymaster.mydomain.com, IP: 1.2.3.4

Ensure that the following entries are present in the Subject Alternative Names for the master’s serving certificate:

EntryExample

Kubernetes service IP address

172.30.0.1

All master host names

master1.example.com

All master IP addresses

192.168.122.1

Public master host name in clustered environments

public-master.example.com

kubernetes

 

kubernetes.default

 

kubernetes.default.svc

 

kubernetes.default.svc.cluster.local

 

openshift

 

openshift.default

 

openshift.default.svc

 

openshift.default.svc.cluster.local

 

If these names are already contained within the Subject Alternative Names, then no further steps are required.

3.2.5.2.2. Generating a New Master Certificate

If your current master certificate does not contain all names from the list above, then you must generate a new certificate for your master:

  1. Back up the existing /etc/origin/master/master.server.crt and /etc/origin/master/master.server.key files for your master: 

    # mv /etc/origin/master/master.server.crt /etc/origin/master/master.server.crt.bak
# mv /etc/origin/master/master.server.key /etc/origin/master/master.server.key.bak

  2. Export the service names. These names will be used when generating the new certificate: 

    # export service_names="kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster.local,openshift,openshift.default,openshift.default.svc,openshift.default.svc.cluster.local"

  3. You will need the first IP in the services subnet (the kubernetes service IP) as well as the values of masterIP, masterURL and publicMasterURL contained in the /etc/origin/master/master-config.yaml file for the following steps.

    The kubernetes service IP can be obtained with: 

    # oc get svc/kubernetes --template='{{.spec.clusterIP}}'

  4. Generate the new certificate: 

    # oadm ca create-master-certs \
      --hostnames=<master_hostnames>,<master_IP_addresses>,<kubernetes_service_IP>,$service_names \ 1 2 3
      --master=<internal_master_address> \ 4
      --public-master=<public_master_address> \ 5
      --cert-dir=/etc/origin/master/ \
      --overwrite=false
    1
    Adjust <master_hostnames> to match your master host name. In a clustered environment, add all master host names.
    2
    Adjust <master_IP_addresses> to match the value of masterIP. In a clustered environment, add all master IP addresses.
    3
    Adjust <kubernetes_service_IP> to the first IP in the kubernetes services subnet.
    4
    Adjust <internal_master_address> to match the value of masterURL.
    5
    Adjust <public_master_address> to match the value of masterPublicURL.

  5. Restart master services. For single master deployments: 

    # systemctl restart atomic-openshift-master

    For native HA multiple master deployments: 

    # systemctl restart atomic-openshift-master-api
# systemctl restart atomic-openshift-master-controllers

    For Pacemaker HA multiple master deployments: 

    # pcs resource restart master

    After the service restarts, the certificate update is complete.

3.2.6. Upgrading the EFK Logging Stack

If you have previously deployed the EFK logging stack and want to upgrade to the latest logging component images, the steps must be performed manually as shown in Manual Upgrades.

3.2.7. Verifying the Upgrade

To verify the upgrade, first check that all nodes are marked as Ready: 

# oc get nodes
NAME                 LABELS                                                                STATUS
master.example.com   kubernetes.io/hostname=master.example.com,region=infra,zone=default   Ready
node1.example.com    kubernetes.io/hostname=node1.example.com,region=primary,zone=east     Ready

Then, verify that you are running the expected versions of the docker-registry and router images, if deployed: 

# oc get -n default dc/docker-registry -o json | grep \"image\"
    "image": "openshift3/ose-docker-registry:v3.1.1.11",
# oc get -n default dc/router -o json | grep \"image\"
    "image": "openshift3/ose-haproxy-router:v3.1.1.11",

If you upgraded from OSE 3.0 to OSE 3.1, verify in your old /etc/sysconfig/openshift-master and /etc/sysconfig/openshift-node files that any custom configuration is added to your new /etc/sysconfig/atomic-openshift-master and /etc/sysconfig/atomic-openshift-node files.

After upgrading, you can use the experimental diagnostics tool to look for common issues: 

# openshift ex diagnostics
...
[Note] Summary of diagnostics execution:
[Note] Completed with no errors or warnings seen.

3.3. Performing Manual Cluster Upgrades

3.3.1. Overview

As an alternative to performing an automated upgrade, you can manually upgrade your OpenShift cluster. To manually upgrade without disruption, it is important to upgrade each component as documented in this topic.

Important

Before you begin your upgrade, familiarize yourself now with the entire procedure. Specific releases may require additional steps to be performed at key points before or during the standard upgrade process.

3.3.2. Preparing for a Manual Upgrade

  1. If you are upgrading from OpenShift Enterprise 3.0 to 3.1, perform the following steps:

    1. On each master and node host, manually disable the 3.0 channel and enable the 3.1 channel: 

      # subscription-manager repos --disable="rhel-7-server-ose-3.0-rpms" \
    --enable="rhel-7-server-ose-3.1-rpms"

    2. Create an etcd backup on each master: 

      # yum install etcd
# etcdctl backup --data-dir /var/lib/openshift/openshift.local.etcd \
    --backup-dir /var/lib/openshift/openshift.local.etcd.bak
    3. Remove support for the v1beta3 API. Update the /etc/openshift/master/master-config.yml file on each master, and remove v1beta3 from the apiLevels and kubernetesMasterConfig.apiLevels parameters.

    4. During this upgrade, some directories are renamed from openshift to origin, so create the following symlinks on each host: 

      # ln -s /var/lib/openshift /var/lib/origin
# ln -s /etc/openshift /etc/origin

  2. Install or update to the following latest available *-excluder packages on each RHEL 7 system, which helps ensure your systems stay on the correct versions of atomic-openshift and docker packages when you are not trying to upgrade, according to the OpenShift Enterprise version: 

    # yum install atomic-openshift-excluder atomic-openshift-docker-excluder

    These packages add entries to the exclude directive in the host’s /etc/yum.conf file.

  3. If you are already running OpenShift Enterprise 3.1 or later, create an etcd backup by running: 

    # yum install etcd
# etcdctl backup --data-dir /var/lib/origin/openshift.local.etcd \
    --backup-dir /var/lib/origin/openshift.local.etcd.bak

  4. For any upgrade path, always ensure that you are running the latest kernel: 

    # yum update kernel

3.3.3. Upgrading Masters

Upgrade your master hosts first:

  1. Run the following command on each master to remove the atomic-openshift packages from the list of yum excludes on the host: 

    # atomic-openshift-excluder unexclude

  2. Upgrade the atomic-openshift packages or related images. 

    # yum upgrade atomic-openshift-master

  3. If you are upgrading from OpenShift Enterprise 3.0 to 3.1:

    1. Create the following master proxy client certificates: 

      # cd /etc/origin/master/
# oadm ca create-master-certs --cert-dir=/etc/origin/master/ \
            --master=https://<internal-master-fqdn>:8443 \
            --public-master=https://<external-master-fqdn>:8443 \
            --hostnames=<external-master-fqdn>,<internal-master-fqdn>,localhost,127.0.0.1,<master-ip-address>,kubernetes.default.local \
            --overwrite=false

      This creates files at /etc/origin/master/master.proxy-client.{crt,key}.

    2. Then, add the master proxy client certificates to the /etc/origin/master/master-config.yml file on each master: 

      kubernetesMasterConfig:
  proxyClientInfo:
    certFile: master.proxy-client.crt
    keyFile: master.proxy-client.key

    3. Enable the following renamed service on master hosts: 

      # systemctl enable atomic-openshift-master

  4. For any upgrade path, now restart the atomic-openshift-master service and review its logs to ensure services have been restarted successfully: 

    # systemctl restart atomic-openshift-master
# journalctl -r -u atomic-openshift-master

  5. Run the following command on each master to add the atomic-openshift packages back to the list of yum excludes on the host: 

    # atomic-openshift-excluder exclude

3.3.4. Updating Policy Definitions

After a cluster upgrade, the recommended default cluster roles may have been updated. To check if an update is recommended for your environment, you can run: 

# oadm policy reconcile-cluster-roles

This command outputs a list of roles that are out of date and their new proposed values. For example: 

# oadm policy reconcile-cluster-roles
apiVersion: v1
items:
- apiVersion: v1
  kind: ClusterRole
  metadata:
    creationTimestamp: null
    name: admin
  rules:
  - attributeRestrictions: null
    resources:
    - builds/custom
...
Note

Your output will vary based on the OpenShift version and any local customizations you have made. Review the proposed policy carefully.

You can either modify this output to re-apply any local policy changes you have made, or you can automatically apply the new policy using the following process:

  1. Reconcile the cluster roles: 

    # oadm policy reconcile-cluster-roles --confirm

  2. Restart the master service: 

    # systemctl restart atomic-openshift-master

  3. Reconcile security context constraints: 

    # oadm policy reconcile-sccs \
    --additive-only=true \
    --confirm

3.3.5. Upgrading Nodes

After upgrading your masters, you can upgrade your nodes. When restarting the atomic-openshift-node service, there will be a brief disruption of outbound network connectivity from running pods to services while the service proxy is restarted. The length of this disruption should be very short and scales based on the number of services in the entire cluster.

To upgrade nodes:

  1. Run the following command on each node to remove the atomic-openshift packages from the list of yum excludes on the host: 

    # atomic-openshift-excluder unexclude

  2. As a user with cluster-admin privileges, disable scheduling for the node: 

    # oadm manage-node <node> --schedulable=false

  3. On each node host, upgrade all atomic-openshift packages: 

    # yum upgrade atomic-openshift\*

  4. If you are upgrading from OpenShift Enterprise 3.0 to 3.1, enable the following renamed service on node hosts: 

    # systemctl enable atomic-openshift-node

  5. For any upgrade path, now restart the atomic-openshift-node service: 

    # systemctl restart atomic-openshift-node

  6. Enable scheduling again for any non-master nodes that you disabled: 

    # oadm manage-node <node> --schedulable=true

  7. Run the following command on the node to add the atomic-openshift packages back to the list of yum excludes on the host: 

    # atomic-openshift-excluder exclude
  8. Repeat these steps on the next node, and continue repeating these steps until all nodes have been upgraded.

  9. After all nodes have been upgraded, as a user with cluster-admin privileges, verify that all nodes are showing as Ready: 

    # oc get nodes
NAME                    LABELS                                        STATUS
master.example.com      kubernetes.io/hostname=master.example.com     Ready,SchedulingDisabled
node1.example.com       kubernetes.io/hostname=node1.example.com      Ready
node2.example.com       kubernetes.io/hostname=node2.example.com      Ready

3.3.6. Upgrading the Router

If you have previously deployed a router, the router deployment configuration must be upgraded to apply updates contained in the router image. To upgrade your router without disrupting services, you must have previously deployed a highly-available routing service.

Edit your router’s deployment configuration. For example, if it has the default router name: 

# oc edit dc/router

Apply the following changes: 

...
spec:
 template:
    spec:
      containers:
      - env:
        ...
        image: registry.access.redhat.com/openshift3/ose-haproxy-router:v3.1.1.11 1
        imagePullPolicy: IfNotPresent
        ...
1
Adjust the image version to match the version you are upgrading to.

You should see one router pod updated and then the next.

3.3.7. Upgrading the Registry

The registry must also be upgraded for changes to take effect in the registry image. If you have used a PersistentVolumeClaim or a host mount point, you may restart the registry without losing the contents of your registry. Deploying a Docker Registry details how to configure persistent storage for the registry.

Edit your registry’s deployment configuration: 

# oc edit dc/docker-registry

Apply the following changes: 

...
spec:
 template:
    spec:
      containers:
      - env:
        ...
        image: registry.access.redhat.com/openshift3/ose-docker-registry:v3.1.1.11 1
        imagePullPolicy: IfNotPresent
        ...
1
Adjust the image version to match the version you are upgrading to.
Important

Images that are being pushed or pulled from the internal registry at the time of upgrade will fail and should be restarted automatically. This will not disrupt pods that are already running.

3.3.8. Updating the Default Image Streams and Templates

By default, the quick and advanced installation methods automatically create default image streams, InstantApp templates, and database service templates in the openshift project, which is a default project to which all users have view access. These objects were created during installation from the JSON files located under the /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/ directory.

Note

Because RHEL Atomic Host 7 cannot use yum to update packages, the following steps must take place on a RHEL 7 system.

  1. Update the packages that provide the example JSON files. On a subscribed Red Hat Enterprise Linux 7 system where you can run the CLI as a user with cluster-admin permissions, install or update to the latest version of the atomic-openshift-utils package, which should also update the openshift-ansible- packages: 

    # yum update atomic-openshift-utils

    The openshift-ansible-roles package provides the latest example JSON files.

  2. Update the global openshift project by running the following commands. Receiving warnings about items that already exist is expected. 

    # oc create -n openshift -f /usr/share/openshift/examples/image-streams/image-streams-rhel7.json
# oc create -n openshift -f /usr/share/openshift/examples/db-templates
# oc create -n openshift -f /usr/share/openshift/examples/quickstart-templates
# oc create -n openshift -f /usr/share/openshift/examples/xpaas-streams
# oc create -n openshift -f /usr/share/openshift/examples/xpaas-templates
# oc replace -n openshift -f /usr/share/openshift/examples/image-streams/image-streams-rhel7.json
# oc replace -n openshift -f /usr/share/openshift/examples/db-templates
# oc replace -n openshift -f /usr/share/openshift/examples/quickstart-templates
# oc replace -n openshift -f /usr/share/openshift/examples/xpaas-streams
# oc replace -n openshift -f /usr/share/openshift/examples/xpaas-templates

  3. After a manual upgrade, get the latest templates from openshift-ansible-roles: 

    rpm -ql openshift-ansible-roles | grep examples | grep v1.2

    In this example, /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/image-streams/image-streams-rhel7.json is the latest file that you want in the latest openshift-ansible-roles package.

    /usr/share/openshift/examples/image-streams/image-streams-rhel7.json is not owned by a package, but is updated by Ansible. If you are upgrading outside of Ansible. you need to get the latest .json files on the system where you are running oc, which can run anywhere that has access to the master.

  4. Install atomic-openshift-utils and its dependencies to install the new content into /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/.: 

    $ oc create -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/image-streams/image-streams-rhel7.json
$ oc replace -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/image-streams/image-streams-rhel7.json

  5. Update the templates: 

    $ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/quickstart-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/db-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/infrastructure-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/xpaas-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/xpaas-streams/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/quickstart-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/db-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/infrastructure-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/xpaas-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/xpaas-streams/

    Errors are generated for items that already exist. This is expected behavior: 

    # oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/quickstart-templates/
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/quickstart-templates/cakephp-mysql.json": templates "cakephp-mysql-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/quickstart-templates/cakephp.json": templates "cakephp-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/quickstart-templates/dancer-mysql.json": templates "dancer-mysql-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/quickstart-templates/dancer.json": templates "dancer-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.2/quickstart-templates/django-postgresql.json": templates "django-psql-example" already exists

Now, content can be updated. Without running the automated upgrade playbooks, the content is not updated in /usr/share/openshift/.

3.3.9. Importing the Latest Images

After updating the default image streams, you may also want to ensure that the images within those streams are updated. For each image stream in the default openshift project, you can run: 

# oc import-image -n openshift <imagestream>

For example, get the list of all image streams in the default openshift project: 

# oc get is -n openshift
NAME     DOCKER REPO                                                      TAGS                   UPDATED
mongodb  registry.access.redhat.com/openshift3/mongodb-24-rhel7           2.4,latest,v3.1    16 hours ago
mysql    registry.access.redhat.com/openshift3/mysql-55-rhel7             5.5,latest,v3.1    16 hours ago
nodejs   registry.access.redhat.com/openshift3/nodejs-010-rhel7           0.10,latest,v3.1   16 hours ago
...

Update each image stream one at a time: 

# oc import-image -n openshift nodejs
Waiting for the import to complete, CTRL+C to stop waiting.
The import completed successfully.

Name:                   nodejs
Created:                16 hours ago
Labels:                 <none>
Annotations:            openshift.io/image.dockerRepositoryCheck=2015-07-21T13:17:00Z
Docker Pull Spec:       registry.access.redhat.com/openshift3/nodejs-010-rhel7

Tag             Spec            Created         PullSpec                                                        Image
0.10            latest          16 hours ago    registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest   66d92cebc0e48e4e4be3a93d0f9bd54f21af7928ceaa384d20800f6e6fcf669f
latest                          16 hours ago    registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest   66d92cebc0e48e4e4be3a93d0f9bd54f21af7928ceaa384d20800f6e6fcf669f
v3.1            <pushed>        16 hours ago    registry.access.redhat.com/openshift3/nodejs-010-rhel7:v3.1     66d92cebc0e48e4e4be3a93d0f9bd54f21af7928ceaa384d20800f6e6fcf669f
Important

In order to update your S2I-based applications, you must manually trigger a new build of those applications after importing the new images using oc start-build <app-name>.

3.3.10. Updating Master and Node Certificates

The following steps may be required for any OpenShift cluster that was originally installed prior to the OpenShift Enterprise 3.1 release. This may include any and all updates from that version.

3.3.10.1. Node Certificates

With the 3.1 release, certificates for each of the kubelet nodes were updated to include the IP address of the node. Any node certificates generated before the 3.1 release may not contain the IP address of the node.

If a node is missing the IP address as part of its certificate, clients may refuse to connect to the kubelet endpoint. Usually this will result in errors regarding the certificate not containing an IP SAN.

In order to remedy this situation, you may need to manually update the certificates for your node.

3.3.10.1.1. Checking the Node’s Certificate

The following command can be used to determine which Subject Alternative Names (SANs) are present in the node’s serving certificate. In this example, the Subject Alternative Names are mynode, mynode.mydomain.com, and 1.2.3.4: 

# openssl x509 -in /etc/origin/node/server.crt -text -noout | grep -A 1 "Subject Alternative Name"
X509v3 Subject Alternative Name:
DNS:mynode, DNS:mynode.mydomain.com, IP: 1.2.3.4

Ensure that the nodeIP value set in the /etc/origin/node/node-config.yaml file is present in the IP values from the Subject Alternative Names listed in the node’s serving certificate. If the nodeIP is not present, then it will need to be added to the node’s certificate.

If the nodeIP value is already contained within the Subject Alternative Names, then no further steps are required.

You will need to know the Subject Alternative Names and nodeIP value for the following steps.

3.3.10.1.2. Generating a New Node Certificate

If your current node certificate does not contain the proper IP address, then you must regenerate a new certificate for your node.

Important

Node certificates will be regenerated on the master (or first master) and are then copied into place on node systems.

  1. Create a temporary directory in which to perform the following steps: 

    # mkdir /tmp/node_certificate_update
# cd /tmp/node_certificate_update

  2. Export the signing options: 

    # export signing_opts="--signer-cert=/etc/origin/master/ca.crt \
    --signer-key=/etc/origin/master/ca.key \
    --signer-serial=/etc/origin/master/ca.serial.txt"

  3. Generate the new certificate: 

    # oadm ca create-server-cert --cert=server.crt \
  --key=server.key $signing_opts \
  --hostnames=<existing_SANs>,<nodeIP>

    For example, if the Subject Alternative Names from before were mynode, mynode.mydomain.com, and 1.2.3.4, and the nodeIP was 10.10.10.1, then you would need to run the following command: 

    # oadm ca create-server-cert --cert=server.crt \
  --key=server.key $signing_opts \
  --hostnames=mynode,mynode.mydomain.com,1.2.3.4,10.10.10.1
3.3.10.1.3. Replace Node Serving Certificates

Back up the existing /etc/origin/node/server.crt and /etc/origin/node/server.key files for your node: 

# mv /etc/origin/node/server.crt /etc/origin/node/server.crt.bak
# mv /etc/origin/node/server.key /etc/origin/node/server.key.bak

You must now copy the new server.crt and server.key created in the temporary directory during the previous step: 

# mv /tmp/node_certificate_update/server.crt /etc/origin/node/server.crt
# mv /tmp/node_certificate_update/server.key /etc/origin/node/server.key

After you have replaced the node’s certificate, restart the node service: 

# systemctl restart atomic-openshift-node

3.3.10.2. Master Certificates

With the 3.1 release, certificates for each of the masters were updated to include all names that pods may use to communicate with masters. Any master certificates generated before the 3.1 release may not contain these additional service names.

3.3.10.2.1. Checking the Master’s Certificate

The following command can be used to determine which Subject Alternative Names (SANs) are present in the master’s serving certificate. In this example, the Subject Alternative Names are mymaster, mymaster.mydomain.com, and 1.2.3.4: 

# openssl x509 -in /etc/origin/master/master.server.crt -text -noout | grep -A 1 "Subject Alternative Name"
X509v3 Subject Alternative Name:
DNS:mymaster, DNS:mymaster.mydomain.com, IP: 1.2.3.4

Ensure that the following entries are present in the Subject Alternative Names for the master’s serving certificate:

EntryExample

Kubernetes service IP address

172.30.0.1

All master host names

master1.example.com

All master IP addresses

192.168.122.1

Public master host name in clustered environments

public-master.example.com

kubernetes

 

kubernetes.default

 

kubernetes.default.svc

 

kubernetes.default.svc.cluster.local

 

openshift

 

openshift.default

 

openshift.default.svc

 

openshift.default.svc.cluster.local

 

If these names are already contained within the Subject Alternative Names, then no further steps are required.

3.3.10.2.2. Generating a New Master Certificate

If your current master certificate does not contain all names from the list above, then you must generate a new certificate for your master:

  1. Back up the existing /etc/origin/master/master.server.crt and /etc/origin/master/master.server.key files for your master: 

    # mv /etc/origin/master/master.server.crt /etc/origin/master/master.server.crt.bak
# mv /etc/origin/master/master.server.key /etc/origin/master/master.server.key.bak

  2. Export the service names. These names will be used when generating the new certificate: 

    # export service_names="kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster.local,openshift,openshift.default,openshift.default.svc,openshift.default.svc.cluster.local"

  3. You will need the first IP in the services subnet (the kubernetes service IP) as well as the values of masterIP, masterURL and publicMasterURL contained in the /etc/origin/master/master-config.yaml file for the following steps.

    The kubernetes service IP can be obtained with: 

    # oc get svc/kubernetes --template='{{.spec.clusterIP}}'

  4. Generate the new certificate: 

    # oadm ca create-master-certs \
      --hostnames=<master_hostnames>,<master_IP_addresses>,<kubernetes_service_IP>,$service_names \ 1 2 3
      --master=<internal_master_address> \ 4
      --public-master=<public_master_address> \ 5
      --cert-dir=/etc/origin/master/ \
      --overwrite=false
    1
    Adjust <master_hostnames> to match your master host name. In a clustered environment, add all master host names.
    2
    Adjust <master_IP_addresses> to match the value of masterIP. In a clustered environment, add all master IP addresses.
    3
    Adjust <kubernetes_service_IP> to the first IP in the kubernetes services subnet.
    4
    Adjust <internal_master_address> to match the value of masterURL.
    5
    Adjust <public_master_address> to match the value of masterPublicURL.

  5. Restart master services. For single master deployments: 

    # systemctl restart atomic-openshift-master

    For native HA multiple master deployments: 

    # systemctl restart atomic-openshift-master-api
# systemctl restart atomic-openshift-master-controllers

    For Pacemaker HA multiple master deployments: 

    # pcs resource restart master

    After the service restarts, the certificate update is complete.

3.3.11. Upgrading the EFK Logging Stack

If you have previously deployed the EFK logging stack and want to upgrade to the latest logging component images, you must take the following steps to safely upgrade with minimal disruption to your log data.

Note

The following steps apply when you want to update to newer OpenShift Enterprise 3.1 logging images, but are not yet fully upgrading your cluster to a later minor or major release of OpenShift Enterprise. The IMAGE_VERSION variable is used in a later step to ensure that you do not accidentally pull the wrong images.

  1. Ensure you are working in the project where the EFK stack was previously deployed, and stay in that project for the remainder of these steps. For example, if the project is named logging: 

    $ oc project logging

  2. Scale down your Fluentd instances to 0: 

    $ oc scale dc/logging-fluentd --replicas=0

    Wait until they have properly terminated. This helps prevent loss of data by giving them time to properly flush their current buffer and send any logs they were processing to Elasticsearch.

  3. Scale down your Kibana instances: 

    $ oc scale dc/logging-kibana --replicas=0

    If you have an operations deployment, also run: 

    $ oc scale dc/logging-kibana-ops --replicas=0

  4. Once your Fluentd and Kibana pods are confirmed to be terminated, you can safely scale down the Elasticsearch pods: 

    $ oc scale dc/logging-es-<unique_name> --replicas=0

    If you have an operations deployment, also run: 

    $ oc scale dc/logging-es-ops-<unique_name> --replicas=0

  5. After your Elasticsearch pods are confirmed to be terminated, pull in the latest EFK images using the same procedure described in Importing the Latest Images, replacing the openshift project with the project where the EFK stack was previously deployed.

    For example, if the project is named logging: 

    $ oc import-image -n logging <imagestream>

    The list of image streams are: 

    logging-auth-proxy
logging-elasticsearch
logging-fluentd
logging-kibana

  6. With the latest images in your repository, you can now rerun the deployer to generate any missing or changed features.

    1. First, ensure that your OAuth client has been deleted: 

      $ oc delete oauthclient --selector logging-infra=support

    2. Then, proceed to follow the same steps as done previously in Deploying the EFK Stack, but ensure that you add IMAGE_VERSION to the list of variables and set it to the appropriate version. For example, for the latest 3.1.1 image: 

      $ oc process logging-deployer-template -n openshift \
           -v IMAGE_VERSION=3.1.1,KIBANA_HOSTNAME=kibana.example.com,ES_CLUSTER_SIZE=1,PUBLIC_MASTER_URL=https://localhost:8443 \
           | oc create -f -

      See Deploying the EFK Stack for the full instructions. After the deployer completes, re-attach your persistent volumes you were using previously.

  7. Next, scale Elasticsearch back up incrementally so that the cluster has time to rebuild.

    1. To begin, scale up to 1: 

      $ oc scale dc/logging-es-<unique_name> --replicas=1

      Follow the logs of the resulting pod to ensure that it is able to recover its indices correctly and that there are no errors: 

      $ oc logs -f <pod_name>

      If that is successful, you can then do the same for the operations cluster, if one was previously used.

    2. After all Elasticsearch nodes have recovered their indices, continue to scale it back up to the size it was prior to doing maintenance. Check the logs of the Elasticsearch members to verify that they have correctly joined the cluster and recovered.
  8. Now scale Kibana and Fluentd back up to their previous state. Because Fluentd was shut down and allowed to push its remaining records to Elasticsearch in the previous steps, it can now pick back up from where it left off with no loss of logs, provided any unread log files are still available on the node.

  9. In the latest version, Kibana will display indices differently now in order to prevent users from being able to access the logs of previously created projects that have been deleted.

    Due to this change, your old logs will not appear automatically. To migrate your old indices to the new format, rerun the deployer with -v MODE=migrate in addition to your prior flags. This should be run while your Elasticsearch cluster is running, as the script must connect to it to make changes.

    Note

    This only impacts non-operations logs. Operations logs will appear the same as in previous versions. There should be minimal performance impact to Elasticsearch while running this and it will not perform an install.

3.3.12. Additional Manual Steps Per Release

Some OpenShift releases may have additional instructions specific to that release that must be performed to fully apply the updates across the cluster. Read through the following sections carefully depending on your upgrade path, as you may be required to perform certain steps at key points during the standard upgrade process described earlier in this topic.

See the OpenShift Enterprise 3.1 Release Notes to review the latest release notes.

3.3.12.1. OpenShift Enterprise 3.1.0

There are no additional manual steps for these releases that are not already mentioned inline during the standard manual upgrade process.

3.3.12.2. OpenShift Enterprise 3.1.1

There was an issue with OpenShift Enterprise 3.1.1 where hosts with host names that resolved to IP addresses that were not local to the host would run into problems with liveness and readiness probes on newly-created HAProxy routers. This was resolved in RHBA-2016:0293 by configuring the probes to use localhost as the hostname for pods with hostPort values.

If you created a router under the affected version, and your liveness or readiness probes unexpectedly fail for your router, then add host: localhost: 

# oc edit dc/router

Apply the following changes: 

spec:
 template:
    spec:
      containers:
      ...
        livenessProbe:
          httpGet:
            host: localhost 1
            path: /healthz
            port: 1936
            scheme: HTTP
          initialDelaySeconds: 10
          timeoutSeconds: 1
        ...
        readinessProbe:
          httpGet:
            host: localhost 2
            path: /healthz
            port: 1936
            scheme: HTTP
          timeoutSeconds: 1
1
Add 'host: localhost' to your liveness probe.
2
Add 'host: localhost' to your readiness probe.

3.3.12.3. OpenShift Enterprise 3.1.1.11

There are no additional manual steps for the upgrade to OpenShift Enterprise 3.1.1.11 that are not already mentioned inline during the standard manual upgrade process.

3.3.13. Verifying the Upgrade

To verify the upgrade, first check that all nodes are marked as Ready: 

# oc get nodes
NAME                 LABELS                                                                STATUS
master.example.com   kubernetes.io/hostname=master.example.com,region=infra,zone=default   Ready
node1.example.com    kubernetes.io/hostname=node1.example.com,region=primary,zone=east     Ready

Then, verify that you are running the expected versions of the docker-registry and router images, if deployed: 

# oc get -n default dc/docker-registry -o json | grep \"image\"
    "image": "openshift3/ose-docker-registry:v3.1.1.11",
# oc get -n default dc/router -o json | grep \"image\"
    "image": "openshift3/ose-haproxy-router:v3.1.1.11",

If you upgraded from OSE 3.0 to OSE 3.1, verify in your old /etc/sysconfig/openshift-master and /etc/sysconfig/openshift-node files that any custom configuration is added to your new /etc/sysconfig/atomic-openshift-master and /etc/sysconfig/atomic-openshift-node files.

After upgrading, you can use the experimental diagnostics tool to look for common issues: 

# openshift ex diagnostics
...
[Note] Summary of diagnostics execution:
[Note] Completed with no errors or warnings seen.

3.4. Upgrading from Pacemaker to Native HA

3.4.1. Overview

If you are using the Pacemaker method for high availability (HA) masters, you can upgrade to the native HA method either using Ansible playbooks or manually. Both methods are described in the following sections.

3.4.2. Using Ansible Playbooks

These steps assume that cluster has been upgraded to OpenShift Enterprise 3.1 using either the manual or automated method.

Warning

Playbooks used for the Pacemaker to native HA upgrade will re-run cluster configuration steps, therefore any settings that are not stored in your inventory file will be overwritten. Back up any configuration files that have been modified since installation before beginning this upgrade.

3.4.2.1. Modifying the Ansible Inventory

Your original Ansible inventory file’s Pacemaker configuration contains a VIP and a cluster host name which should resolve to this VIP. Native HA requires a cluster host name which resolves to the load balancer being used.

Consider the following example configuration: 

# Pacemaker high availability cluster method.
# Pacemaker HA environment must be able to self provision the
# configured VIP. For installation openshift_master_cluster_hostname
# must resolve to the configured VIP.
#openshift_master_cluster_method=pacemaker
#openshift_master_cluster_password=openshift_cluster
#openshift_master_cluster_vip=192.168.133.35
#openshift_master_cluster_public_vip=192.168.133.35
#openshift_master_cluster_hostname=openshift-cluster.example.com
#openshift_master_cluster_public_hostname=openshift-cluster.example.com

Remove or comment the above section in your inventory file. Then, add the following section, modifying the host names to match your current cluster host names: 

# Native high availability cluster method with optional load balancer.
# If no lb group is defined, the installer assumes that a load balancer has
# been preconfigured. For installation the value of
# openshift_master_cluster_hostname must resolve to the load balancer
# or to one or all of the masters defined in the inventory if no load
# balancer is present.
openshift_master_cluster_method=native
openshift_master_cluster_hostname=openshift-cluster.example.com
openshift_master_cluster_public_hostname=openshift-cluster.example.com

Native HA requires a load balancer to balance the master API (port 8443). When modifying your inventory file, specify an [lb] group and add lb to the [OSEv3:children] section if you would like the playbooks to configure an HAProxy instance as the load balancer. This instance must be on a separate host from the masters and nodes: 

[OSEv3:children]
masters
nodes
lb
...
[lb]
lb1.example.com 1
1
Host name of the HAProxy load balancer.

Any external load balancer may be used in place of the default HAProxy host, but it must be pre-configured and allow API traffic to masters on port 8443.

3.4.2.1.1. Destroying the Pacemaker Cluster

On any master, run the following to destroy the Pacemaker cluster.

Warning

After the Pacemaker cluster has been destroyed, the OpenShift cluster will be in outage until the remaining steps are completed. 

# pcs cluster destroy --all

3.4.2.2. Updating DNS

Modify your cluster host name DNS entries such that the host name used resolves to the load balancer that will be used with native HA.

In the earlier example configuration, openshift-cluster.example.com resolves to 192.168.133.35. DNS must be modified such that openshift-cluster.example.com now resolves to the load balancer host or to the master API balancer in an alternative load balancing solution.

3.4.2.3. Running the Ansible Playbook

You can now run the following Ansible playbook:

Warning

Back up any configuration files that have been modified since installation before beginning this upgrade. 

# ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml

After the playbook finishes successfully, your upgrade to the native HA method is complete. Restore any configuration files if you backed up any that had been modified since installation, and restart any relevant OpenShift services, if necessary.

3.4.3. Manually Upgrading

These steps assume that cluster has been upgraded to OpenShift Enterprise 3.1 using either the manual or automated method. They also assume that you are bringing your own load balancer which has been configured to balance the master API on port 8443.

3.4.3.1. Creating Unit and System Configuration for New Services

The Systemd unit files for the atomic-openshift-master-api and atomic-openshift-master-controllers services are not yet provided by packaging. Ansible creates unit and system configuration when installing with the native HA method.

Therefore, create the following files:

Example 3.1. /usr/lib/systemd/system/atomic-openshift-master-api.service File

[Unit]
Description=Atomic OpenShift Master API
Documentation=https://github.com/openshift/origin
After=network.target
After=etcd.service
Before=atomic-openshift-node.service
Requires=network.target

[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/atomic-openshift-master-api
Environment=GOTRACEBACK=crash
ExecStart=/usr/bin/openshift start master api --config=${CONFIG_FILE} $OPTIONS
LimitNOFILE=131072
LimitCORE=infinity
WorkingDirectory=/var/lib/origin/
SyslogIdentifier=atomic-openshift-master-api

[Install]
WantedBy=multi-user.target
WantedBy=atomic-openshift-node.service

Example 3.2. /usr/lib/systemd/system/atomic-openshift-master-controllers.service File

[Unit]
Description=Atomic OpenShift Master Controllers
Documentation=https://github.com/openshift/origin
After=network.target
After=atomic-openshift-master-api.service
Before=atomic-openshift-node.service
Requires=network.target

[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/atomic-openshift-master-controllers
Environment=GOTRACEBACK=crash
ExecStart=/usr/bin/openshift start master controllers --config=${CONFIG_FILE} $OPTIONS
LimitNOFILE=131072
LimitCORE=infinity
WorkingDirectory=/var/lib/origin/
SyslogIdentifier=atomic-openshift-master-controllers
Restart=on-failure

[Install]
WantedBy=multi-user.target
WantedBy=atomic-openshift-node.service

Example 3.3. /etc/sysconfig/atomic-openshift-master-api File

OPTIONS=--loglevel=2
CONFIG_FILE=/etc/origin/master/master-config.yaml

# Proxy configuration
# Origin uses standard HTTP_PROXY environment variables. Be sure to set
# NO_PROXY for your master
#NO_PROXY=master.example.com
#HTTP_PROXY=http://USER:PASSWORD@IPADDR:PORT
#HTTPS_PROXY=https://USER:PASSWORD@IPADDR:PORT

Example 3.4. /etc/sysconfig/atomic-openshift-master-controllers File

OPTIONS=--loglevel=2
CONFIG_FILE=/etc/origin/master/master-config.yaml

# Proxy configuration
# Origin uses standard HTTP_PROXY environment variables. Be sure to set
# NO_PROXY for your master
#NO_PROXY=master.example.com
#HTTP_PROXY=http://USER:PASSWORD@IPADDR:PORT
#HTTPS_PROXY=https://USER:PASSWORD@IPADDR:PORT

Then, reload Systemd to pick up your changes: 

# systemctl daemon-reload

3.4.3.2. Destroying the Pacemaker Cluster

On any master, run the following to destroy the Pacemaker cluster.

Warning

After the Pacemaker cluster has been destroyed, the OpenShift cluster will be in outage until the remaining steps are completed. 

# pcs cluster destroy --all

3.4.3.3. Updating DNS

Modify your cluster host name DNS entries such that the host name used resolves to the load balancer that will be used with native HA.

For example, if the cluster host name is openshift-cluster.example.com and it resolved to a VIP of 192.168.133.35, then DNS must be modified such that openshift-cluster.example.com now resolves to the master API balancer.

3.4.3.4. Modifying Master and Node Configuration

Edit the master configuration in the /etc/origin/master/master-config.yaml file and ensure that kubernetesMasterConfig.masterCount is updated to the total number of masters. Perform this step on all masters:

Example 3.5. /etc/origin/master/master-config.yaml File

...
kubernetesMasterConfig:
  apiServerArguments:
    controllerArguments:
      masterCount: 3 1
...
1
Update this value to the total number of masters.

Edit the node configuration in the /etc/orign/node/node-config.yaml file and remove the dnsIP setting. OpenShift will use the Kubernetes service IP as the dnsIP by default. Perform this step on all nodes:

Example 3.6. /etc/origin/node/node-config.yaml File

...
allowDisabledDocker: false
apiVersion: v1
dnsDomain: cluster.local
dnsIP: 10.6.102.3 1
dockerConfig:
  execHandlerName: ""
...
1
Remove this line.
3.4.3.4.1. Starting the API Service

Start and enable the API service on all masters: 

# systemctl start atomic-openshift-master-api
# systemctl enable atomic-openshift-master-api
3.4.3.4.2. Starting the Controller Service

Start and enable the controllers service on all masters: 

# systemctl start atomic-openshift-master-controllers
# systemctl enable atomic-openshift-master-controllers

After the service restarts, your upgrade to the native HA method is complete.

3.4.3.5. Modifying the Ansible Inventory

Optionally, modify your Ansible inventory file for future runs per the instructions above in the playbooks method.