Procedure 4.1. To Prepare OpenShift Enterprise for an Upgrade:

1. Perform the required backup steps before starting with the upgrade. Only proceed to the next step after the backup is complete, and the relevant personnel are notified of the upcoming outage.
2. Disable any change management software that is being used to manage your OpenShift Enterprise installation configuration, and update it accordingly after the upgrade.
3. If a configuration file already exists on disk during an update, the RPM package that provides the file does one of the following, depending on how the package is built:
   • Backs up the existing file with an .rpmsave extension and creates the new file.
   • Leaves the existing file in place and creates the new file with an .rpmnew extension.
   Before updating, find any .rpm* files still on disk from previous updates using the following commands:

   # updatedb
   # locate --regex '\.rpm(save|new)$'

   Compare these files to the relevant configuration files currently in use and note any differences. Manually merge any desired settings into the current configuration files, then either move the .rpm* files to an archive directory or remove them.
4. Before attempting to upgrade, ensure the latest errata have been applied for the current minor version of your OpenShift Enterprise installation. Run the yum update command, then check again for any new configuration files that have changed:

   # yum update -y
   # updatedb
   # locate --regex '\.rpm(save|new)$'

   Resolve any .rpm* files found again as described in the previous step.
   Additional steps may also be required depending on the errata being applied. For more information on errata updates, see the relevant OpenShift Enterprise Release Notes at http://access.redhat.com/site/documentation.
5. Restart any services that had their configuration files updated.
6. Run the oo-admin-chk script on a broker host:

   # oo-admin-chk

   This command checks the integrity of the MongoDB datastore against the actual deployment of application gears on the node hosts. Resolve any issues reported by this script, if possible, prior to performing an upgrade. For more information on using the oo-admin-chk script and fixing gear discrepancies, see the OpenShift Enterprise Troubleshooting Guide at http://access.redhat.com/site/documentation.
7. Run the oo-diagnostics script on all hosts:

   # oo-diagnostics

   Use the output of this command to compare after the upgrade is complete.

Procedure 4.2. To Bootstrap the Upgrade and Perform the begin Step:

1. The openshift-enterprise-release RPM package includes the ose-upgrade tool that guides you through the upgrade process. Install the openshift-enterprise-release package on each host, and update it to the most current version.

   # yum install openshift-enterprise-release

2. The begin step of the upgrade process applies to all hosts, and includes those hosts that contain only supporting services such as MongoDB and ActiveMQ. Hosts using Red Hat Subscription Management (RHSM) or Red Hat Network (RHN) Classic are unsubscribed from the 1.2 channels and subscribed to the new 2.0 channels.

   Warning

   This step assumes that the channel names come directly from Red Hat Network. If the package source is an instance of Red Hat Satellite or Subscription Asset Manager and the channel names are remapped differently, you must change this yourself. Examine the scripts in the /usr/lib/ruby/site_ruby/1.8/ose-upgrade/host/upgrades/2/ directory for use as models. You can also add your custom script to a subdirectory to be executed with the ose-upgrade tool.
   In addition to updating the channel set, modifications to the yum configuration give priority to the OpenShift Enterprise, Red Hat Enterprise Linux, and JBoss repositories. However, packages from other sources are excluded as required to prevent certain issues with dependency management that occur between the various channels.
   Run the begin step on each host. Note that the command output is different depending on the type of host. The following example output is from a broker host:

   # ose-upgrade begin
   
   INFO: OpenShift broker installed.
   INFO: Setting host step 'begin' status to UPGRADING
   INFO: Starting upgrade number 2 to version 2.0.
   [...]
   INFO: Setting host step 'begin' status to COMPLETE
   INFO: To continue the upgrade, install a specific upgrade package.

Procedure 4.3. To Install the Upgrade RPM Specific to a Host:

1. Depending on the host type, install the latest upgrade RPM package from the new OpenShift Enterprise 2.0 channels. For broker hosts, install the openshift-enterprise-upgrade-broker package:

   # yum install openshift-enterprise-upgrade-broker

   For node hosts, install the openshift-enterprise-upgrade-node package:

   # yum install openshift-enterprise-upgrade-node

   If the package is already installed because of a previous upgrade, it still must be updated to the latest package version for the OpenShift Enterprise 2.0 upgrade.
2. The ose-upgrade tool guides the upgrade process by listing the necessary steps that are specific to the upgrade scenario, and identifies the step to be performed next. The ose-upgrade status command, or ose-upgrade, provides a current status report. The command output varies depending on the type of host. The following example output is from a broker host:

   # ose-upgrade status
   
   INFO: OpenShift broker installed.
   Current upgrade is number 2 to version 2.0.
   Step sequence:
     begin pre outage rpms conf maintenance_mode pending_ops confirm_nodes data gears end_maintenance_mode post
   Next step is: pre

Procedure 4.4. To Perform the pre Step on Broker and Node Hosts:

1. The pre step manages the following actions:
   • Backs up OpenShift Enterprise configuration files.
   • Clears pending operations older than one hour. (Broker hosts only)
   • Performs any pre-upgrade datastore migration steps. (Broker hosts only)
   • Updates authorization indexes. (Broker hosts only)
   Run the pre step on one broker host and each node host:

   # ose-upgrade pre

   When one broker host begins this step, any attempts made by other broker hosts to run the pre step simultaneously will fail.
2. After the pre step completes on the first broker host, run it on any remaining broker hosts.

Procedure 4.5. To Perform the outage Step on Broker and Node Hosts:

1. The outage step stops services as required depending on the type of host.

   Warning

   The broker enters outage mode during this upgrade step. A substantial outage also begins for applications on the node hosts. Scaled applications are unable to contact any child gears during the outage. These outages last until the end_maintenance_mode step is complete.
   Perform this step on all broker hosts first, and then on all node hosts. This begins the broker outage, and all communication between the broker host and the node hosts is stopped. Perform the outage step with the following command:

   # ose-upgrade outage

   After the command completes on all hosts, node and broker hosts can be upgraded simultaneously until the upgrade steps are complete on all node hosts, and the broker host reaches the confirm_nodes step.
2. For all other hosts that are not a broker or a node host, run yum update to upgrade any services that are installed, such as MongoDB or ActiveMQ:

   # yum update

Procedure 4.6. To Perform the rpms Step on Broker and Node Hosts:

• The rpms step updates RPM packages installed on the node host, and installs any new RPM packages that are required.
  Run the rpms step on each host:

  # ose-upgrade rpms

Procedure 4.7. To Perform the conf Step on Broker and Node Hosts:

• The conf step changes the OpenShift Enterprise configuration to match the new codebase installed in the previous step. Each modified file is first copied to a file with the same name plus a .ugsave extension and a timestamp. This makes it easier to determine what files have changed.
  Run the conf step on each host:

  # ose-upgrade conf

  Warning

  If the configuration files have been significantly modified from the recommended configuration, manual intervention may be required to merge configuration changes so that they can be used with OpenShift Enterprise.

Procedure 4.8. To Perform the maintenance_mode Step on Broker and Node Hosts:

• The maintenance_mode step manages the following actions:
  • Configures the broker to disable the API and return an outage notification to any requests. (Broker hosts only)
  • Starts the broker service and, if installed, the console service in maintenance mode so that they provide clients with an outage notification. (Broker hosts only)
  • Clears the broker and console caches. (Broker hosts only)
  • Enables gear upgrade extensions. (Node hosts only)
  • Starts the ruby193-mcollective service. (Node hosts only)
  Run the maintenance_mode step on each host:

  # ose-upgrade maintenance_mode

Procedure 4.9. To Perform the pending_ops Step on a Broker Host:

1. The pending_ops step clears records of any pending application operations; the outage prevents them from ever completing. Run the pending_ops step on one broker host. Do not run this command on multiple broker hosts at the same time. When one broker host begins this step, any attempts made by other broker hosts to run the pending_ops step simultaneously will fail:

   # ose-upgrade pending_ops

2. After the pending_ops step completes on the first broker host, run the command on any remaining broker hosts.

Procedure 4.10. To Perform the confirm_nodes Step on Broker Hosts:

1. The confirm_nodes step attempts to access all known node hosts to determine whether they have all been upgraded before proceeding. This step fails if the maintenance_mode step has not been completed on all node hosts, or if MCollective cannot access any node hosts.
   Run the confirm_nodes step on a broker host:

   # ose-upgrade confirm_nodes

2. If this step fails due to node hosts that are no longer deployed, you may need to skip the confirm_nodes step. Ensure that all node hosts reported missing are not actually expected to respond, then skip the confirm_nodes step with the following command:

   # ose-upgrade --skip confirm_nodes

Procedure 4.11. To Perform the data Step on Broker Hosts:

1. The data step runs a data migration against the shared broker datastore. Run the data step on one broker host:

   # ose-upgrade data

   When one broker host begins this step, any attempts made by other broker hosts to run the data step simultaneously will fail.
2. After the data step completes on the first broker host, run it on any remaining broker hosts.

Procedure 4.12. To Perform the gears Step on Broker Hosts:

1. The gears step runs a gear migration through the required changes so that they can be used in OpenShift Enterprise 2.0. Run the gears step on one broker host:

   # ose-upgrade gears

   When one broker host begins this step, any attempts made by other broker hosts to run the gears step simultaneously will fail.
2. After the gears step completes on the first broker host, run it on any remaining broker hosts.

Procedure 4.13. To Perform the test_gears_complete Step on Node Hosts:

• The test_gears_complete step verifies the gear migrations are complete before proceeding. This step blocks the upgrade on node hosts by waiting until the gears step has completed on an associated broker host. Run the test_gears_complete step on all node hosts:

  # ose-upgrade test_gears_complete

Procedure 4.14. To Perform the end_maintenance_mode Step on Broker and Node Hosts:

1. The end_maintenance_mode step starts the services that were stopped in the maintenance_mode step or added in the interim. It gracefully restarts httpd to complete the node host upgrade, and restarts the broker service and, if installed, the console service. Complete this step on all node hosts first before running it on the broker hosts:

   # ose-upgrade end_maintenance_mode

2. Run the oo-accept-node script on each node host to verify that it is correctly configured:

   # oo-accept-node

Procedure 4.15. To Perform the post Step on Broker Hosts:

1. The post step manages the following actions on the broker host:
   • Performs any post-upgrade datastore migration steps.
   • Publishes updated district UIDs to the node hosts.
   • Clears the broker and console caches.
   Run the post step on a broker host:

   # ose-upgrade post

   When one broker host begins this step, any attempts made by other broker hosts to run the post step simultaneously will fail.
2. After the post step completes on the first broker host, run it on any remaining broker hosts.
3. The upgrade is now complete for an OpenShift Enterprise installation. Run oo-diagnostics on each host to diagnose any problems:

   # oo-diagnostics

Procedure 4.16. To Bootstrap the Upgrade and Perform the begin Step:

1. The openshift-enterprise-release RPM package includes the ose-upgrade tool that guides you through the upgrade process. Install the openshift-enterprise-release package on each host, and update it to the most current version.

   # yum install openshift-enterprise-release

2. The begin step of the upgrade process applies to all hosts, and includes those hosts that contain only supporting services such as MongoDB and ActiveMQ. Hosts using Red Hat Subscription Management (RHSM) or Red Hat Network (RHN) Classic are unsubscribed from the 2.0 channels and subscribed to the new 2.1 channels.

   Warning

   This step assumes that the channel names come directly from Red Hat Network. If the package source is an instance of Red Hat Satellite or Subscription Asset Manager and the channel names are remapped differently, you must change this yourself. Examine the scripts in the /usr/lib/ruby/site_ruby/1.8/ose-upgrade/host/upgrades/3/ directory for use as models. You can also add your custom script to a subdirectory to be executed with the ose-upgrade tool.
   In addition to updating the channel set, modifications to the yum configuration give priority to the OpenShift Enterprise, Red Hat Enterprise Linux, and JBoss repositories. However, packages from other sources are excluded as required to prevent certain issues with dependency management that occur between the various channels.
   Run the begin step on each host. Note that the command output is different depending on the type of host. The following example output is from a broker host:

   # ose-upgrade begin
   
   INFO: OpenShift broker installed.
   INFO: Setting host step 'begin' status to UPGRADING
   INFO: Starting upgrade number 3 to version 2.1.
   [...]
   INFO: updating /etc/openshift-enterprise-release
   INFO: Setting host step 'begin' status to COMPLETE
   INFO: To continue the upgrade, install a specific upgrade package.

   Important

   The oo-admin-yum-validator --oo-version 2.1 --fix-all command is run automatically during the begin step. When using RHN Classic, the command does not automatically subscribe a system to the OpenShift Enterprise 2.1 channels, but instead reports the manual steps required. After the channels are manually subscribed, running the begin step again sets the proper yum priorities and continues as expected.

Procedure 4.17. To Install the Upgrade RPM Specific to a Host:

1. Depending on the host type, install the latest upgrade RPM package from the new OpenShift Enterprise 2.1 channels. For broker hosts, install the openshift-enterprise-upgrade-broker package:

   # yum install openshift-enterprise-upgrade-broker

   For node hosts, install the openshift-enterprise-upgrade-node package:

   # yum install openshift-enterprise-upgrade-node

   If the package is already installed because of a previous upgrade, it still must be updated to the latest package version for the OpenShift Enterprise 2.1 upgrade.
2. The ose-upgrade tool guides the upgrade process by listing the necessary steps that are specific to the upgrade scenario, and identifies the step to be performed next. The ose-upgrade status command, or ose-upgrade, provides a current status report. The command output varies depending on the type of host. The following example output is from a broker host:

   # ose-upgrade status
   
   INFO: OpenShift broker installed.
   Current upgrade is number 3 to version 2.1.
   Step sequence:
     begin pre outage rpms conf maintenance_mode pending_ops confirm_nodes data gears end_maintenance_mode post
   Next step is: pre

Procedure 4.18. To Perform the pre Step on Broker and Node Hosts:

1. The pre step manages the following actions:
   • Backs up OpenShift Enterprise configuration files.
   • Clears pending operations older than one hour. (Broker hosts only)
   • Performs any pre-upgrade datastore migration steps. (Broker hosts only)
   Run the pre step on one broker host and each node host:

   # ose-upgrade pre

   When one broker host begins this step, any attempts made by other broker hosts to run the pre step simultaneously will fail.
2. After the pre step completes on the first broker host, run it on any remaining broker hosts.

Procedure 4.19. To Perform the outage Step on Broker and Node Hosts:

1. The outage step stops services as required depending on the type of host.

   Warning

   The broker enters outage mode during this upgrade step. A substantial outage also begins for applications on the node hosts. Scaled applications are unable to contact any child gears during the outage. These outages last until the end_maintenance_mode step is complete.
   Perform this step on all broker hosts first, and then on all node hosts. This begins the broker outage, and all communication between the broker host and the node hosts is stopped. Perform the outage step with the following command:

   # ose-upgrade outage

   After the command completes on all hosts, node and broker hosts can be upgraded simultaneously until the upgrade steps are complete on all node hosts, and the broker host reaches the confirm_nodes step.
2. For all other hosts that are not a broker or a node host, run yum update to upgrade any services that are installed, such as MongoDB or ActiveMQ:

   # yum update

Procedure 4.20. To Perform the rpms Step on Broker and Node Hosts:

• The rpms step updates RPM packages installed on the host, and installs any new RPM packages that are required. For node hosts, this includes the recommended cartridge dependency metapackages for any cartridge already installed on a node. See Section 9.8.3, “Installing Cartridge Dependency Metapackages” for more information about cartridge dependency metapackages.
  Run the rpms step on each host:

  # ose-upgrade rpms

Procedure 4.21. To Perform the conf Step on Broker and Node Hosts:

• The conf step changes the OpenShift Enterprise configuration to match the new codebase installed in the previous step. Each modified file is first copied to a file with the same name plus a .ugsave extension and a timestamp. This makes it easier to determine what files have changed.
  Run the conf step on each host:

  # ose-upgrade conf

  Warning

  If the configuration files have been significantly modified from the recommended configuration, manual intervention may be required to merge configuration changes so that they can be used with OpenShift Enterprise.

Procedure 4.22. To Perform the maintenance_mode Step on Broker and Node Hosts:

• The maintenance_mode step manages the following actions:
  • Configures the broker to disable the API and return an outage notification to any requests. (Broker hosts only)
  • Starts the broker service and, if installed, the console service in maintenance mode so that they provide clients with an outage notification. (Broker hosts only)
  • Clears the broker and console caches. (Broker hosts only)
  • Enables gear upgrade extensions. (Node hosts only)
  • Saves and regenerates configurations for any apache-vhost front ends. (Node hosts only)
  • Stops the openshift-iptables-port-proxy service. (Node hosts only)
  • Starts the ruby193-mcollective service. (Node hosts only)
  Run the maintenance_mode step on each host:

  # ose-upgrade maintenance_mode

Procedure 4.23. To Perform the pending_ops Step on Broker Hosts:

1. The pending_ops step clears records of any pending application operations because the outage prevents them from ever completing. Run the pending_ops step on one broker host only:

   # ose-upgrade pending_ops

2. On any remaining broker hosts, run the following command to skip the pending_ops step:

   # ose-upgrade pending_ops --skip

Procedure 4.24. To Perform the confirm_nodes Step on Broker Hosts:

1. The confirm_nodes step attempts to access all known node hosts to determine whether they have all been upgraded before proceeding. This step fails if the maintenance_mode step has not been completed on all node hosts, or if MCollective cannot access any node hosts.
   Run the confirm_nodes step on a broker host:

   # ose-upgrade confirm_nodes

2. If this step fails due to node hosts that are no longer deployed, you may need to skip the confirm_nodes step. Ensure that all node hosts reported missing are not actually expected to respond, then skip the confirm_nodes step with the following command:

   # ose-upgrade --skip confirm_nodes

Procedure 4.25. To Perform the data Step on Broker Hosts:

1. The data step runs a data migration against the shared broker datastore. Run the data step on one broker host:

   # ose-upgrade data

   When one broker host begins this step, any attempts made by other broker hosts to run the data step simultaneously will fail.
2. After the data step completes on the first broker host, run it on any remaining broker hosts.

Procedure 4.26. To Perform the gears Step on Broker Hosts:

1. The gears step runs a gear migration through the required changes so that they can be used in OpenShift Enterprise 2.1. Run the gears step on one broker host:

   # ose-upgrade gears

   When one broker host begins this step, any attempts made by other broker hosts to run the gears step simultaneously will fail.
2. After the gears step completes on the first broker host, run it on any remaining broker hosts.

Procedure 4.27. To Perform the test_gears_complete Step on Node Hosts:

• The test_gears_complete step verifies the gear migrations are complete before proceeding. This step blocks the upgrade on node hosts by waiting until the gears step has completed on an associated broker host. Run the test_gears_complete step on all node hosts:

  # ose-upgrade test_gears_complete

Procedure 4.28. To Perform the end_maintenance_mode Step on Broker and Node Hosts:

1. The end_maintenance_mode step starts the services that were stopped in the maintenance_mode step or added in the interim. It gracefully restarts httpd to complete the node host upgrade, and restarts the broker service and, if installed, the console service. Complete this step on all node hosts first before running it on the broker hosts:

   # ose-upgrade end_maintenance_mode

2. Run the oo-accept-node script on each node host to verify that it is correctly configured:

   # oo-accept-node

Procedure 4.29. To Perform the post Step on Broker Hosts:

1. The post step manages the following actions on the broker host:
   • Imports cartridges to the datastore.
   • Performs any post-upgrade datastore migration steps.
   • Clears the broker and console caches.
   Run the post step on a broker host:

   # ose-upgrade post

   When one broker host begins this step, any attempts made by other broker hosts to run the post step simultaneously will fail.
2. After the post step completes on the first broker host, run it on any remaining broker hosts.
3. The upgrade is now complete for an OpenShift Enterprise installation. Run oo-diagnostics on each host to diagnose any problems:

   # oo-diagnostics

Procedure 4.30. To Bootstrap the Upgrade and Perform the begin Step:

1. The openshift-enterprise-release RPM package includes the ose-upgrade tool that guides you through the upgrade process. Install the openshift-enterprise-release package on each host, and update it to the most current version.

   # yum install openshift-enterprise-release

2. The begin step of the upgrade process applies to all hosts, and includes those hosts that contain only supporting services such as MongoDB and ActiveMQ. Hosts using Red Hat Subscription Management (RHSM) or Red Hat Network (RHN) Classic are unsubscribed from the 2.1 channels and subscribed to the new 2.2 channels.

   Warning

   This step assumes that the channel names come directly from Red Hat Network. If the package source is an instance of Red Hat Satellite or Subscription Asset Manager and the channel names are remapped differently, you must change this yourself. Examine the scripts in the /usr/lib/ruby/site_ruby/1.8/ose-upgrade/host/upgrades/4/ directory for use as models. You can also add your custom script to a subdirectory to be executed with the ose-upgrade tool.
   In addition to updating the channel set, modifications to the yum configuration give priority to the OpenShift Enterprise, Red Hat Enterprise Linux, and JBoss repositories. However, packages from other sources are excluded as required to prevent certain issues with dependency management that occur between the various channels.
   Run the begin step on each host. Note that the command output is different depending on the type of host. The following example output is from a broker host:

   # ose-upgrade begin
   
   INFO: OpenShift broker installed.
   INFO: Setting host step 'begin' status to UPGRADING
   INFO: Starting upgrade number 4 to version 2.2.
   [...]
   INFO: updating /etc/openshift-enterprise-release
   INFO: Setting host step 'begin' status to COMPLETE
   INFO: To continue the upgrade, install a specific upgrade package.

   Important

   The oo-admin-yum-validator --oo-version 2.2 --fix-all command is run automatically during the begin step. When using RHN Classic, the command does not automatically subscribe a system to the OpenShift Enterprise 2.2 channels, but instead reports the manual steps required. After the channels are manually subscribed, running the begin step again sets the proper yum priorities and continues as expected.

Procedure 4.31. To Install the Upgrade RPM Specific to a Host:

1. Depending on the host type, install the latest upgrade RPM package from the new OpenShift Enterprise 2.2 channels. For broker hosts, install the openshift-enterprise-upgrade-broker package:

   # yum install openshift-enterprise-upgrade-broker

   For node hosts, install the openshift-enterprise-upgrade-node package:

   # yum install openshift-enterprise-upgrade-node

   If the package is already installed because of a previous upgrade, it still must be updated to the latest package version for the OpenShift Enterprise 2.2 upgrade.
2. The ose-upgrade tool guides the upgrade process by listing the necessary steps that are specific to the upgrade scenario, and identifies the step to be performed next. The ose-upgrade status command, or ose-upgrade, provides a current status report. The command output varies depending on the type of host. The following example output is from a broker host:

   # ose-upgrade status
   
   INFO: OpenShift broker installed.
   Current upgrade is number 4 to version 2.2.
   Step sequence:
     begin pre outage rpms conf maintenance_mode pending_ops confirm_nodes data gears end_maintenance_mode post
   Next step is: pre

Procedure 4.32. To Perform the pre Step on Broker and Node Hosts:

1. The pre step manages the following actions:
   • Backs up OpenShift Enterprise configuration files.
   • Clears pending operations older than one hour. (Broker hosts only)
   • Performs any pre-upgrade datastore migration steps. (Broker hosts only)
   Run the pre step on one broker host and each node host:

   # ose-upgrade pre

   When one broker host begins this step, any attempts made by other broker hosts to run the pre step simultaneously will fail.
2. After the pre step completes on the first broker host, run it on any remaining broker hosts.
3. After the pre step completes on all hosts, the ose-upgrade tool allows you to continue through the node and broker host upgrade steps in parallel. On broker hosts, the tool will block the confirm_nodes step if the associated node hosts have not completed their maintenance_mode step. On node hosts, the tool blocks the test_gears_complete step if the associated broker has not completed the gears step.
   Continue through the following procedures for instructions on each subsequent step.

Procedure 4.33. To Perform the rpms Step on Broker and Node Hosts:

1. The rpms step updates RPM packages installed on the host and installs any new RPM packages that are required. For node hosts, this includes the recommended cartridge dependency metapackages for any cartridge already installed on a node. See Section 9.8.3, “Installing Cartridge Dependency Metapackages” for more information about cartridge dependency metapackages.
   Run the rpms step on each host:

   # ose-upgrade rpms

2. For all other hosts that are not a broker or a node host, run yum update to upgrade any services that are installed, such as MongoDB or ActiveMQ:

   # yum update

Procedure 4.34. To Perform the conf Step on Broker and Node Hosts:

• The conf step changes the OpenShift Enterprise configuration to match the new codebase installed in the previous step. Each modified file is first copied to a file with the same name plus a .ugsave extension and a timestamp. This makes it easier to determine what files have changed.
  This step also disables the SSLv3 protocol on each broker host in favor of TLS due to CVE-2014-3566.
  Run the conf step on each host:

  # ose-upgrade conf

  Warning

  If the configuration files have been significantly modified from the recommended configuration, manual intervention may be required to merge configuration changes so that they can be used with OpenShift Enterprise.

Procedure 4.35. To Perform the maintenance_mode Step on Broker and Node Hosts:

1. Starting with OpenShift Enterprise 2.2, the apache-mod-rewrite front-end server proxy plug-in is deprecated. New deployments of OpenShift Enterprise 2.2 now use the apache-vhost plug-in as the default.

   Important

   Any new nodes added to your deployment after the upgrade will use the apache-vhost plug-in by default. Note that the apache-mod-rewrite plug-in is incompatible with the apache-vhost plug-in, and the front-end server configuration on all nodes across a deployment must be consistent. See Section 10.1, “Front-End Server Proxies” for more information.
   The default behavior of the maintenance_mode step is to leave the apache-mod-rewrite plug-in in place, if it is installed. Do not set the OSE_UPGRADE_MIGRATE_VHOST environment variable at all, not even to false or 0, if you require this default behavior.
   However, if your OpenShift Enterprise 2.1 deployment was configured to use the apache-mod-rewrite plug-in before starting the 2.2 upgrade, you can optionally allow the ose-upgrade tool to migrate your node hosts to the newly-default apache-vhost plug-in. To enable this option, set the OSE_UPGRADE_MIGRATE_VHOST environment variable on each node host:

   # export OSE_UPGRADE_MIGRATE_VHOST=true

2. The maintenance_mode step manages actions in the following order:
   • Configures the broker to disable the API and return an outage notification to any requests. (Broker hosts only)
   • Restarts the broker service and, if installed, the console service in maintenance mode so that they provide clients with an outage notification. (Broker hosts only)
   • Clears the broker and console caches. (Broker hosts only)
   • Stops the ruby193-mcollective service. (Node hosts only)
   • Saves the front-end server proxy configuration. (Node hosts only)
   • If the OSE_UPGRADE_MIGRATE_VHOST environment variable was set in the previous step, migrates from the apache-mod-rewrite plug-in to the apache-vhost plug-in. (Node hosts only)
   • Disables the SSLv3 protocol in favor of TLS due to CVE-2014-3566. (Node hosts only)
   • Enables gear upgrade extensions. (Node hosts only)
   • Starts the ruby193-mcollective service. (Node hosts only)
   Run the maintenance_mode step on each host:

   # ose-upgrade maintenance_mode

Procedure 4.36. To Perform the pending_ops Step on Broker Hosts:

1. The pending_ops step clears records of any pending application operations because the outage prevents them from ever completing. Run the pending_ops step on one broker host:

   # ose-upgrade pending_ops

   When one broker host begins this step, any attempts made by other broker hosts to run the pending_ops step simultaneously will fail.
2. After the pending_ops step completes on the first broker host, run it on any remaining broker hosts.

Procedure 4.37. To Perform the confirm_nodes Step on Broker Hosts:

1. The confirm_nodes step attempts to access all known node hosts to determine whether they have all been upgraded before proceeding. This step fails if the maintenance_mode step has not been completed on all node hosts, or if MCollective cannot access any node hosts.
   Run the confirm_nodes step on a broker host:

   # ose-upgrade confirm_nodes

2. If this step fails due to node hosts that are no longer deployed, you may need to skip the confirm_nodes step. Ensure that all node hosts reported missing are not actually expected to respond, then skip the confirm_nodes step with the following command:

   # ose-upgrade --skip confirm_nodes

Procedure 4.38. To Perform the data Step on Broker Hosts:

1. The data step runs a data migration against the shared broker datastore. Run the data step on one broker host:

   # ose-upgrade data

   When one broker host begins this step, any attempts made by other broker hosts to run the data step simultaneously will fail.
2. After the data step completes on the first broker host, run it on any remaining broker hosts.

Procedure 4.39. To Perform the gears Step on Broker Hosts:

1. The gears step runs a gear migration through the required changes so that they can be used in OpenShift Enterprise 2.2. Run the gears step on one broker host:

   # ose-upgrade gears

   When one broker host begins this step, any attempts made by other broker hosts to run the gears step simultaneously will fail.
2. After the gears step completes on the first broker host, run it on any remaining broker hosts.

Procedure 4.40. To Perform the test_gears_complete Step on Node Hosts:

• The test_gears_complete step verifies the gear migrations are complete before proceeding. This step blocks the upgrade on node hosts by waiting until the gears step has completed on an associated broker host. Run the test_gears_complete step on all node hosts:

  # ose-upgrade test_gears_complete

Procedure 4.41. To Perform the end_maintenance_mode Step on Broker and Node Hosts:

1. The end_maintenance_mode step restarts the following services on the node hosts:
   • httpd (Restarts gracefully)
   • ruby193-mcollective
   • openshift-iptables-port-proxy
   • openshift-node-web-proxy
   • openshift-sni-proxy
   • openshift-watchman
   Complete this step on all node hosts first before running it on the broker hosts:

   # ose-upgrade end_maintenance_mode

2. After the end_maintenance_mode command has completed on all node hosts, run the same command on the broker hosts to disable the outage notification enabled during the broker maintenance_mode step and restart the broker service and, if installed, the console service:

   # ose-upgrade end_maintenance_mode

   This allows the broker to respond to client requests normally again.
3. Run the oo-accept-node script on each node host to verify that it is correctly configured:

   # oo-accept-node

Procedure 4.42. To Perform the post Step on Broker Hosts:

1. The post step manages the following actions on the broker host:
   • Imports cartridges to the datastore.
   • Performs any post-upgrade datastore migration steps.
   • Clears the broker and console caches.
   Run the post step on a broker host:

   # ose-upgrade post

   When one broker host begins this step, any attempts made by other broker hosts to run the post step simultaneously will fail.
2. After the post step completes on the first broker host, run it on any remaining broker hosts.
3. The upgrade is now complete for an OpenShift Enterprise installation. Run oo-diagnostics on each host to diagnose any problems:

   # oo-diagnostics

Procedure 6.1. To Run the Installation Utility From the Internet:

1. You can run the installation utility directly from the Internet with the following command:

   $ sh <(curl -s https://install.openshift.com/ose-2.2)

   Additional options can be used with the command. These options are outlined later in this section:

   $ sh <(curl -s https://install.openshift.com/ose-2.2) -s rhsm -u user@company.com

2. Follow the on-screen instructions to either deploy a new OpenShift Enterprise host, or add a node host to an existing deployment.

Procedure 6.2. To Download and Run the Installation Utility:

1. Download and unpack the installation utility:

   $ curl -o oo-install-ose.tgz https://install.openshift.com/portable/oo-install-ose.tgz
   $ tar -zxf oo-install-ose.tgz

2. Execute the installation utility to interactively configure one or more hosts:

   $ ./oo-install-ose

   The oo-install-ose utility automatically runs the installation utility in OpenShift Enterprise mode. Additional options can be used with the command. These options are outlined later in this section:

   $ ./oo-install-ose -s rhsm -u user@company.com

3. Follow the on-screen instructions to either deploy a new OpenShift Enterprise host, or add a node host to an existing deployment.

Procedure 7.3. To Configure Yum on Broker Hosts:

1. Install the latest openshift-enterprise-release package:

   # yum install openshift-enterprise-release

2. Run the oo-admin-yum-validator command with the -o option for version 2.2 and the -r option for the broker role. This reports the first detected set of problems, provides a set of proposed changes, and halts.

   Example 7.3. Detecting Problems

   # oo-admin-yum-validator -o 2.2 -r broker
   				
   Please note: --role=broker implicitly enables --role=client to ensure /usr/bin/rhc is available for testing and troubleshooting.
   Detected OpenShift Enterprise repository subscription managed by Red Hat Subscription Manager.
   
   The required OpenShift Enterprise repositories are disabled:
       rhel-server-rhscl-6-rpms
       rhel-6-server-ose-2.2-rhc-rpms
       rhel-6-server-rpms
   Enable these repositories by running these commands:
   # subscription-manager repos --enable=rhel-server-rhscl-6-rpms
   # subscription-manager repos --enable=rhel-6-server-ose-2.2-rhc-rpms
   # subscription-manager repos --enable=rhel-6-server-rpms
   Please re-run this tool after making any recommended repairs to this system

   Alternatively, use the --report-all option to report all detected problems.

   # oo-admin-yum-validator -o 2.2 -r broker --report-all

3. After reviewing the reported problems and their proposed changes, either fix them manually or let the tool attempt to fix the first set of problems using the same command with the --fix option. This may require several repeats of steps 2 and 3.

   Example 7.4. Fixing Problems

   # oo-admin-yum-validator -o 2.2 -r broker --fix
   
   Please note: --role=broker implicitly enables --role=client to ensure /usr/bin/rhc is available for testing and troubleshooting.
   Detected OpenShift Enterprise repository subscription managed by Red Hat Subscription Manager.
   
   Enabled repository rhel-server-rhscl-6-rpms
   Enabled repository rhel-6-server-ose-2.2-rhc-rpms
   Enabled repository rhel-6-server-rpms

   Alternatively, use the --fix-all option to allow the tool to attempt to fix all of the problems that are found.

   # oo-admin-yum-validator -o 2.2 -r broker --fix-all

   Note

   If the host is using Red Hat Network (RHN) Classic, the --fix and --fix-all options do not automatically enable any missing OpenShift Enterprise channels as they do when the host is using Red Hat Subscription Management. Enable the recommended channels with the rhn-channel command. Replace repo-id in the following command with the repository ID reported in the oo-admin-yum-validator command output.

   # rhn-channel -a -c repo-id

   Important

   For either subscription method, the --fix and --fix-all options do not automatically install any packages. The tool reports if any manual steps are required.
4. Repeat steps 2 and 3 until the oo-admin-yum-validator command displays the following message.

   No problems could be detected!

Procedure 8.15. To Install and Configure the Gear Placement Plug-in:

1. Install the gear placement plug-in on each broker host:

   # yum install rubygem-openshift-origin-gear-placement

   This installs a gem with a Rails engine containing the GearPlacementPlugin class.
2. On each broker host, copy the /etc/openshift/plugins.d/openshift-origin-gear-placement.conf.example file to /etc/openshift/plugins.d/openshift-origin-gear-placement.conf:

   # cp /etc/openshift/plugins.d/openshift-origin-gear-placement.conf.example /etc/openshift/plugins.d/openshift-origin-gear-placement.conf

   As long as this configuration file with a .conf extension exists, the broker automatically loads a gem matching the file name, and the gem can use the file to configure itself.
3. Restart the broker service:

   # service openshift-broker restart

   If you make further modifications to the configuration file of the gear placement plug-in, you must restart the broker service again after making your final changes.
4. The default implementation of the plug-in simply logs the plug-in inputs and delegates the actual gear placement to the default algorithm. You can verify that the plug-in is correctly installed and configured with the default implementation by creating an application and checking the /var/log/openshift/broker/production.log file.

   Example 8.7. Checking Broker Logs for Default Gear Placement Plug-in Activity

   2014-10-17 12:53:18.476 [INFO ] Parameters: {"cartridges"=>["php-5.4"], "scale"=>true, "name"=>"mytestapp", "domain_id"=>"demo"} (pid:14508)
   2014-10-17 12:53:24.715 [INFO ] Using gear placement plugin to choose node. (pid:14508)
   2014-10-17 12:53:24.715 [INFO ] selecting from nodes: node2.example.com, node1.example.com (pid:14508)
   2014-10-17 12:53:24.718 [INFO ] server_infos: [#<NodeProperties:0x00000007675438
     @district_available_capacity=5994,
     @district_id="5441e3896892df06a4000001",
     @name="node2.example.com",
     @node_consumed_capacity=3.3333333333333335,
     @region_id=nil,
     @zone_id=nil>,
    #<NodeProperties:0x00000006e302a0
     @district_available_capacity=5994,
     @district_id="5441e3896892df06a4000001",
     @name="node1.example.com",
     @node_consumed_capacity=6.666666666666667,
     @region_id=nil,
     @zone_id=nil>] (pid:14508)
   2014-10-17 12:53:24.719 [INFO ] app_props: #<ApplicationProperties:0x000000078b97a8
    @id="54446b0e6892dff4b5000001",
    @name="mytestapp",
    @web_cartridge="php-5.4"> (pid:14508)
   2014-10-17 12:53:24.720 [INFO ] current_gears: [] (pid:14508)
   2014-10-17 12:53:24.721 [INFO ] comp_list: [#<ComponentProperties:0x000000076a8b08
     @cartridge_name="haproxy-1.4",
     @cartridge_vendor="redhat",
     @component_name="web_proxy",
     @version="1.4">,
    #<ComponentProperties:0x000000076a88d8
     @cartridge_name="php-5.4",
     @cartridge_vendor="redhat",
     @component_name="php-5.4",
     @version="5.4">] (pid:14508)
   2014-10-17 12:53:24.724 [INFO ] user_props: #<UserProperties:0x000000078b8f38
    @capabilities=
     {"ha"=>false,
      "subaccounts"=>false,
      "gear_sizes"=>["small"],
      "max_domains"=>10,
      "max_gears"=>100,
      "max_teams"=>0,
      "view_global_teams"=>false,
      "max_storage_per_gear"=>0},
    @consumed_gears=7,
    @id="5441e5f26892df39e9000001",
    @login="demo",
    @plan_id=nil,
    @plan_state=nil> (pid:14508)
   2014-10-17 12:53:24.724 [INFO ] selected node: 'node2.example.com' (pid:14508)
   

Procedure 8.21. To Install and Configure the SSO Plug-in:

1. On the broker host, install the rubygem-openshift-origin-sso-activemq package:

   # yum install rubygem-openshift-origin-sso-activemq

2. Before enabling this plug-in, you must add a new user, topic, and queue to ActiveMQ. Edit the /etc/activemq/activemq.xml file and add the following user in the appropriate section:

   <authenticationUser username="ssoinfo" password="ssoinfopasswd" groups="ssoinfo,everyone"/>

   Also add the following topic and queue to the appropriate sections:

   <authorizationEntry topic="ssoinfo.>" write="ssoinfo" read="ssoinfo" admin="ssoinfo" />
   <authorizationEntry queue="ssoinfo.>" write="ssoinfo" read="ssoinfo" admin="ssoinfo" />

3. Restart ActiveMQ:

   # service activemq restart

4. To enable the plug-in, copy the /etc/openshift/plugins.d/openshift-origin-sso-activemq.conf.example file to /etc/openshift/plugins.d/openshift-origin-sso-activemq.conf on the broker host:

   # cp /etc/openshift/plugins.d/openshift-origin-sso-activemq.conf.example \ /etc/openshift/plugins.d/openshift-origin-sso-activemq.conf

5. In the /etc/openshift/plugins.d/openshift-origin-sso-activemq.conf file you just created, uncomment the last line specifying the /opt/rh/ruby193/root/etc/mcollective/client.cfg file:

   MCOLLECTIVE_CONFIG="/opt/rh/ruby193/root/etc/mcollective/client.cfg"

   Alternatively, edit the values for the ACTIVE_* parameters with the appropriate information for your environment.
6. Restart the broker service for your changes take effect:

   # service openshift-broker restart

7. Create a listener that will connect to ActiveMQ on the new topic that was added. The listener can be run on any system that can connect to the ActiveMQ server. The following is an example that simply echoes any messages received:

   #!/usr/bin/ruby
   
   require 'rubygems'
   require 'stomp'
   require 'yaml'
   
   c = Stomp::Client.new("ssoinfo", "ssoinfopasswd", "127.0.0.1", 61613)
   puts "Got stomp client, listening for messages on '/topic/ssoinfo':"
   c.subscribe('/topic/ssoinfo') { |msg|
           h = YAML.load(msg.body)
           puts "Message received: "
   	puts h.inspect
   }
   c.join

8. Save and run your listener script. For example, if the script was saved at /root/listener.rb:

   # ruby /root/listener.rb

9. To verify that the plug-in and listener are working, perform several application actions with the client tools or Management Console using a test user account. For example, create an application, add an alias, remove an alias, and remove the application. You should see messages reported by the listener script for each action performed.

Procedure 9.3. To Configure Yum on Node Hosts:

1. Install the latest openshift-enterprise-release package:

   # yum install openshift-enterprise-release

2. Run the oo-admin-yum-validator command with the -o option for version 2.2 and the -r option for the node role.
   If you intend to install one or more xPaaS premium cartridge and the relevant subscription(s) are in place as described in Section 9.1, “Configuring Node Host Entitlements”, replace node with one or more of the node-eap, node-amq, or node-fuse roles as needed for the respective cartridge(s). If you add more than one role, use an -r option when defining each role.
   The command reports the first detected set of problems, provides a set of proposed changes, and halts.

   Example 9.5. Finding Problems

   # oo-admin-yum-validator -o 2.2 -r node-eap
   				
   Detected OpenShift Enterprise repository subscription managed by Red Hat Subscription Manager.
   
   The required OpenShift Enterprise repositories are disabled:
       jb-ews-2-for-rhel-6-server-rpms
       rhel-6-server-rpms
       rhel-6-server-ose-2.2-jbosseap-rpms
       rhel-server-rhscl-6-rpms
       jb-eap-6-for-rhel-6-server-rpms
   Enable these repositories by running these commands:
   # subscription-manager repos --enable=jb-ews-2-for-rhel-6-server-rpms
   # subscription-manager repos --enable=rhel-6-server-rpms
   # subscription-manager repos --enable=rhel-6-server-ose-2.2-jbosseap-rpms
   # subscription-manager repos --enable=rhel-server-rhscl-6-rpms
   # subscription-manager repos --enable=jb-eap-6-for-rhel-6-server-rpms
   Please re-run this tool after making any recommended repairs to this system

   Alternatively, use the --report-all option to report all detected problems.

   # oo-admin-yum-validator -o 2.2 -r node-eap --report-all

3. After reviewing the reported problems and their proposed changes, either fix them manually or let the tool attempt to fix the first set of problems using the same command with the --fix option. This may require several repeats of steps 2 and 3.

   Example 9.6. Fixing Problems

   # oo-admin-yum-validator -o 2.2 -r node-eap --fix
   
   Detected OpenShift Enterprise repository subscription managed by Red Hat Subscription Manager.
   
   Enabled repository jb-ews-2-for-rhel-6-server-rpms
   Enabled repository rhel-6-server-rpms
   Enabled repository rhel-6-server-ose-2.2-jbosseap-rpms
   Enabled repository rhel-server-rhscl-6-rpms
   Enabled repository jb-eap-6-for-rhel-6-server-rpms

   Alternatively, use the --fix-all option to allow the tool to attempt to fix all of the problems that are found.

   # oo-admin-yum-validator -o 2.2 -r node-eap --fix-all

   Note

   If the host is using Red Hat Network (RHN) Classic, the --fix and --fix-all options do not automatically enable any missing OpenShift Enterprise channels as they do when the host is using Red Hat Subscription Management. Enable the recommended channels with the rhn-channel command. Replace repo-id in the following command with the repository ID reported in the oo-admin-yum-validator command output.

   # rhn-channel -a -c repo-id

   Important

   For either subscription method, the --fix and --fix-all options do not automatically install any packages. The tool reports if any manual steps are required.
4. Repeat steps 2 and 3 until the oo-admin-yum-validator command displays the following message.

   No problems could be detected!

Procedure 9.4. To Configure the DHCP Client and Host Name on the Node Host:

1. Create the /etc/dhcp/dhclient-eth0.conf file, then add the following lines to configure the DHCP client to send DNS requests to the broker (Host 1) and assume the appropriate host name and domain name. Replace 10.0.0.1 with the actual IP address of Host 1 and example.com with the actual domain name of Host 2. If you are using a network interface other than eth0, edit the configuration file for that interface instead.

   prepend domain-name-servers 10.0.0.1;
   prepend domain-search "example.com";

2. Edit the /etc/sysconfig/network file on Host 2, and set the HOSTNAME parameter to the fully-qualified domain name (FQDN) of Host 2. Replace node.example.com in the following example with the host name of Host 2.

   HOSTNAME=node.example.com

   Important

   Red Hat does not recommend changing the node host name after the initial configuration. When an application is created on a node host, application data is stored in a database. If the node host name is modified, the data does not automatically change, which can cause the instance to fail. The node host name cannot be changed without deleting and recreating all gears on the node host. Therefore, verify that the host name is configured correctly before deploying any applications on a node host.
3. Set the host name immediately:

   # hostname node.example.com

   Note

   If you use the kickstart or bash script, the configure_dns_resolution and configure_hostname functions perform these steps.
4. Run the hostname command on Host 2:

   # hostname

Procedure 9.5. To Install and Configure MCollective on the Node Host:

1. Install all required packages for MCollective on Host 2 with the following command:

   # yum install openshift-origin-msg-node-mcollective

2. Replace the contents of the /opt/rh/ruby193/root/etc/mcollective/server.cfg file with the following configuration. Remember to change the setting for plugin.activemq.pool.1.host from broker.example.com to the host name of Host 1. Use the same password for the MCollective user specified in the /etc/activemq/activemq.xml file on Host 1. Use the same password for the plugin.psk parameter, and the same numbers for the heartbeat parameters specified in the /opt/rh/ruby193/root/etc/mcollective/client.cfg file on Host 1:

   main_collective = mcollective
   collectives = mcollective
   libdir = /opt/rh/ruby193/root/usr/libexec/mcollective
   logfile = /var/log/openshift/node/ruby193-mcollective.log
   loglevel = debug
   
   daemonize = 1
   direct_addressing = 0
   
   # Plugins
   securityprovider = psk
   plugin.psk = asimplething
   
   connector = activemq
   plugin.activemq.pool.size = 1
   plugin.activemq.pool.1.host = broker.example.com
   plugin.activemq.pool.1.port = 61613
   plugin.activemq.pool.1.user = mcollective
   plugin.activemq.pool.1.password = marionette
   
   plugin.activemq.heartbeat_interval = 30
   plugin.activemq.max_hbread_fails = 2
   plugin.activemq.max_hbrlck_fails = 2
   
   # Node should retry connecting to ActiveMQ forever
   plugin.activemq.max_reconnect_attempts = 0
   plugin.activemq.initial_reconnect_delay = 0.1
   plugin.activemq.max_reconnect_delay = 4.0
   
   # Facts
   factsource = yaml
   plugin.yaml = /opt/rh/ruby193/root/etc/mcollective/facts.yaml

3. Configure the ruby193-mcollective service to start on boot:

   # chkconfig ruby193-mcollective on

4. Start the ruby193-mcollective service immediately:

   # service ruby193-mcollective start

   Note

   If you use the kickstart or bash script, the configure_mcollective_for_activemq_on_node function performs these steps.
5. Run the following command on the broker host (Host 1) to verify that Host 1 recognizes Host 2:

   # oo-mco ping