Procedure 4.30. To Bootstrap the Upgrade and Perform the begin Step:

1. The openshift-enterprise-release RPM package includes the ose-upgrade tool that guides you through the upgrade process. Install the openshift-enterprise-release package on each host, and update it to the most current version.

   # yum install openshift-enterprise-release

2. The begin step of the upgrade process applies to all hosts, and includes those hosts that contain only supporting services such as MongoDB and ActiveMQ. Hosts using Red Hat Subscription Management (RHSM) or Red Hat Network (RHN) Classic are unsubscribed from the 2.1 channels and subscribed to the new 2.2 channels.

   Warning

   This step assumes that the channel names come directly from Red Hat Network. If the package source is an instance of Red Hat Satellite or Subscription Asset Manager and the channel names are remapped differently, you must change this yourself. Examine the scripts in the /usr/lib/ruby/site_ruby/1.8/ose-upgrade/host/upgrades/4/ directory for use as models. You can also add your custom script to a subdirectory to be executed with the ose-upgrade tool.
   In addition to updating the channel set, modifications to the yum configuration give priority to the OpenShift Enterprise, Red Hat Enterprise Linux, and JBoss repositories. However, packages from other sources are excluded as required to prevent certain issues with dependency management that occur between the various channels.
   Run the begin step on each host. Note that the command output is different depending on the type of host. The following example output is from a broker host:

   # ose-upgrade begin
   
   INFO: OpenShift broker installed.
   INFO: Setting host step 'begin' status to UPGRADING
   INFO: Starting upgrade number 4 to version 2.2.
   [...]
   INFO: updating /etc/openshift-enterprise-release
   INFO: Setting host step 'begin' status to COMPLETE
   INFO: To continue the upgrade, install a specific upgrade package.

   Important

   The oo-admin-yum-validator --oo-version 2.2 --fix-all command is run automatically during the begin step. When using RHN Classic, the command does not automatically subscribe a system to the OpenShift Enterprise 2.2 channels, but instead reports the manual steps required. After the channels are manually subscribed, running the begin step again sets the proper yum priorities and continues as expected.

Procedure 4.31. To Install the Upgrade RPM Specific to a Host:

1. Depending on the host type, install the latest upgrade RPM package from the new OpenShift Enterprise 2.2 channels. For broker hosts, install the openshift-enterprise-upgrade-broker package:

   # yum install openshift-enterprise-upgrade-broker

   For node hosts, install the openshift-enterprise-upgrade-node package:

   # yum install openshift-enterprise-upgrade-node

   If the package is already installed because of a previous upgrade, it still must be updated to the latest package version for the OpenShift Enterprise 2.2 upgrade.
2. The ose-upgrade tool guides the upgrade process by listing the necessary steps that are specific to the upgrade scenario, and identifies the step to be performed next. The ose-upgrade status command, or ose-upgrade, provides a current status report. The command output varies depending on the type of host. The following example output is from a broker host:

   # ose-upgrade status
   
   INFO: OpenShift broker installed.
   Current upgrade is number 4 to version 2.2.
   Step sequence:
     begin pre outage rpms conf maintenance_mode pending_ops confirm_nodes data gears end_maintenance_mode post
   Next step is: pre

Procedure 4.32. To Perform the pre Step on Broker and Node Hosts:

1. The pre step manages the following actions:
   • Backs up OpenShift Enterprise configuration files.
   • Clears pending operations older than one hour. (Broker hosts only)
   • Performs any pre-upgrade datastore migration steps. (Broker hosts only)
   Run the pre step on one broker host and each node host:

   # ose-upgrade pre

   When one broker host begins this step, any attempts made by other broker hosts to run the pre step simultaneously will fail.
2. After the pre step completes on the first broker host, run it on any remaining broker hosts.
3. After the pre step completes on all hosts, the ose-upgrade tool allows you to continue through the node and broker host upgrade steps in parallel. On broker hosts, the tool will block the confirm_nodes step if the associated node hosts have not completed their maintenance_mode step. On node hosts, the tool blocks the test_gears_complete step if the associated broker has not completed the gears step.
   Continue through the following procedures for instructions on each subsequent step.

Procedure 4.33. To Perform the rpms Step on Broker and Node Hosts:

1. The rpms step updates RPM packages installed on the host and installs any new RPM packages that are required. For node hosts, this includes the recommended cartridge dependency metapackages for any cartridge already installed on a node. See Section 9.8.3, “Installing Cartridge Dependency Metapackages” for more information about cartridge dependency metapackages.
   Run the rpms step on each host:

   # ose-upgrade rpms

2. For all other hosts that are not a broker or a node host, run yum update to upgrade any services that are installed, such as MongoDB or ActiveMQ:

   # yum update

Procedure 4.34. To Perform the conf Step on Broker and Node Hosts:

• The conf step changes the OpenShift Enterprise configuration to match the new codebase installed in the previous step. Each modified file is first copied to a file with the same name plus a .ugsave extension and a timestamp. This makes it easier to determine what files have changed.
  This step also disables the SSLv3 protocol on each broker host in favor of TLS due to CVE-2014-3566.
  Run the conf step on each host:

  # ose-upgrade conf

  Warning

  If the configuration files have been significantly modified from the recommended configuration, manual intervention may be required to merge configuration changes so that they can be used with OpenShift Enterprise.

Procedure 4.35. To Perform the maintenance_mode Step on Broker and Node Hosts:

1. Starting with OpenShift Enterprise 2.2, the apache-mod-rewrite front-end server proxy plug-in is deprecated. New deployments of OpenShift Enterprise 2.2 now use the apache-vhost plug-in as the default.

   Important

   Any new nodes added to your deployment after the upgrade will use the apache-vhost plug-in by default. Note that the apache-mod-rewrite plug-in is incompatible with the apache-vhost plug-in, and the front-end server configuration on all nodes across a deployment must be consistent. See Section 10.1, “Front-End Server Proxies” for more information.
   The default behavior of the maintenance_mode step is to leave the apache-mod-rewrite plug-in in place, if it is installed. Do not set the OSE_UPGRADE_MIGRATE_VHOST environment variable at all, not even to false or 0, if you require this default behavior.
   However, if your OpenShift Enterprise 2.1 deployment was configured to use the apache-mod-rewrite plug-in before starting the 2.2 upgrade, you can optionally allow the ose-upgrade tool to migrate your node hosts to the newly-default apache-vhost plug-in. To enable this option, set the OSE_UPGRADE_MIGRATE_VHOST environment variable on each node host:

   # export OSE_UPGRADE_MIGRATE_VHOST=true

2. The maintenance_mode step manages actions in the following order:
   • Configures the broker to disable the API and return an outage notification to any requests. (Broker hosts only)
   • Restarts the broker service and, if installed, the console service in maintenance mode so that they provide clients with an outage notification. (Broker hosts only)
   • Clears the broker and console caches. (Broker hosts only)
   • Stops the ruby193-mcollective service. (Node hosts only)
   • Saves the front-end server proxy configuration. (Node hosts only)
   • If the OSE_UPGRADE_MIGRATE_VHOST environment variable was set in the previous step, migrates from the apache-mod-rewrite plug-in to the apache-vhost plug-in. (Node hosts only)
   • Disables the SSLv3 protocol in favor of TLS due to CVE-2014-3566. (Node hosts only)
   • Enables gear upgrade extensions. (Node hosts only)
   • Starts the ruby193-mcollective service. (Node hosts only)
   Run the maintenance_mode step on each host:

   # ose-upgrade maintenance_mode

Procedure 4.36. To Perform the pending_ops Step on Broker Hosts:

1. The pending_ops step clears records of any pending application operations because the outage prevents them from ever completing. Run the pending_ops step on one broker host:

   # ose-upgrade pending_ops

   When one broker host begins this step, any attempts made by other broker hosts to run the pending_ops step simultaneously will fail.
2. After the pending_ops step completes on the first broker host, run it on any remaining broker hosts.

Procedure 4.37. To Perform the confirm_nodes Step on Broker Hosts:

1. The confirm_nodes step attempts to access all known node hosts to determine whether they have all been upgraded before proceeding. This step fails if the maintenance_mode step has not been completed on all node hosts, or if MCollective cannot access any node hosts.
   Run the confirm_nodes step on a broker host:

   # ose-upgrade confirm_nodes

2. If this step fails due to node hosts that are no longer deployed, you may need to skip the confirm_nodes step. Ensure that all node hosts reported missing are not actually expected to respond, then skip the confirm_nodes step with the following command:

   # ose-upgrade --skip confirm_nodes

Procedure 4.38. To Perform the data Step on Broker Hosts:

1. The data step runs a data migration against the shared broker datastore. Run the data step on one broker host:

   # ose-upgrade data

   When one broker host begins this step, any attempts made by other broker hosts to run the data step simultaneously will fail.
2. After the data step completes on the first broker host, run it on any remaining broker hosts.

Procedure 4.39. To Perform the gears Step on Broker Hosts:

1. The gears step runs a gear migration through the required changes so that they can be used in OpenShift Enterprise 2.2. Run the gears step on one broker host:

   # ose-upgrade gears

   When one broker host begins this step, any attempts made by other broker hosts to run the gears step simultaneously will fail.
2. After the gears step completes on the first broker host, run it on any remaining broker hosts.

Procedure 4.40. To Perform the test_gears_complete Step on Node Hosts:

• The test_gears_complete step verifies the gear migrations are complete before proceeding. This step blocks the upgrade on node hosts by waiting until the gears step has completed on an associated broker host. Run the test_gears_complete step on all node hosts:

  # ose-upgrade test_gears_complete

Procedure 4.41. To Perform the end_maintenance_mode Step on Broker and Node Hosts:

1. The end_maintenance_mode step restarts the following services on the node hosts:
   • httpd (Restarts gracefully)
   • ruby193-mcollective
   • openshift-iptables-port-proxy
   • openshift-node-web-proxy
   • openshift-sni-proxy
   • openshift-watchman
   Complete this step on all node hosts first before running it on the broker hosts:

   # ose-upgrade end_maintenance_mode

2. After the end_maintenance_mode command has completed on all node hosts, run the same command on the broker hosts to disable the outage notification enabled during the broker maintenance_mode step and restart the broker service and, if installed, the console service:

   # ose-upgrade end_maintenance_mode

   This allows the broker to respond to client requests normally again.
3. Run the oo-accept-node script on each node host to verify that it is correctly configured:

   # oo-accept-node

Procedure 4.42. To Perform the post Step on Broker Hosts:

1. The post step manages the following actions on the broker host:
   • Imports cartridges to the datastore.
   • Performs any post-upgrade datastore migration steps.
   • Clears the broker and console caches.
   Run the post step on a broker host:

   # ose-upgrade post

   When one broker host begins this step, any attempts made by other broker hosts to run the post step simultaneously will fail.
2. After the post step completes on the first broker host, run it on any remaining broker hosts.
3. The upgrade is now complete for an OpenShift Enterprise installation. Run oo-diagnostics on each host to diagnose any problems:

   # oo-diagnostics