Red Hat Training

A Red Hat training course is available for Red Hat Virtualization

Chapter 8. Upgrading the Self-Hosted Engine

This chapter explains how to upgrade your current environment to Red Hat Virtualization 4.2.

Select the appropriate instructions for your environment from the following table. If your Manager and host versions differ (if you have previously upgraded the Manager but not the hosts), follow the instructions that match the Manager’s version.

For interactive upgrade instructions, you can also use the RHV Upgrade Helper available at https://access.redhat.com/labs/rhvupgradehelper/. This application asks you to provide information about your upgrade path and your current environment, and presents the relevant steps for upgrade as well as steps to prevent known issues specific to your upgrade scenario.

8.1. Upgrading a Self-Hosted Engine from 3.6 to Red Hat Virtualization 4.2

Important

Use these instructions if your environment uses any Next Generation RHVH or Red Hat Enterprise Linux hosts.

If your environment uses only legacy RHEV-H 3.6 hosts, you must upgrade using the instructions in Appendix A, Upgrading a RHEV-H 3.6 Self-Hosted Engine to a RHVH 4.2 Self-Hosted Engine.

You cannot upgrade the Manager directly from 3.6 to 4.2. You must upgrade your environment in the following sequence:

8.1.1. Enabling Global Maintenance Mode

You must place the self-hosted engine environment in global maintenance mode before performing any setup or upgrade tasks on the Manager virtual machine.

Procedure

  1. Log in to one of the self-hosted engine nodes and enable global maintenance mode:

    # hosted-engine --set-maintenance --mode=global
  2. Confirm that the environment is in maintenance mode before proceeding:

    # hosted-engine --vm-status

8.1.2. Updating the Red Hat Virtualization Manager

Updates to the Red Hat Virtualization Manager are released through the Content Delivery Network.

Procedure

  1. On the Red Hat Virtualization Manager machine, check if updated packages are available:

    # engine-upgrade-check
  2. Update the setup packages:

    # yum update rhevm-setup
  3. Update the Red Hat Virtualization Manager. The engine-setup script prompts you with some configuration questions, then stops the ovirt-engine service, downloads and installs the updated packages, backs up and updates the database, performs post-installation configuration, and starts the ovirt-engine service.

    # engine-setup
    Note

    The engine-setup script is also used during the Red Hat Virtualization Manager installation process, and it stores the configuration values supplied. During an update, the stored values are displayed when previewing the configuration, and may not be up to date if engine-config was used to update configuration after installation. For example, if engine-config was used to update SANWipeAfterDelete to true after installation, engine-setup will output "Default SAN wipe after delete: False" in the configuration preview. However, the updated values will not be overwritten by engine-setup.

    Important

    The update process may take some time; allow time for the update process to complete and do not stop the process once initiated.

  4. Update the base operating system and any optional packages installed on the Manager:

    # yum update
    Important

    If any kernel packages were updated, reboot the host to complete the update.

8.1.3. Upgrading the Self-hosted Engine from 3.6 to 4.0

In Red Hat Enterprise Virtualization 3.6, the Manager runs on Red Hat Enterprise Linux 6. An in-place upgrade of the Manager virtual machine to Red Hat Enterprise Linux 7 is not supported.

To upgrade a Red Hat Enterprise Virtualization 3.6 self-hosted engine environment to Red Hat Virtualization 4.0, you must use the upgrade utility that is provided with Red Hat Virtualization 4.0 to install a new Manager on Red Hat Enterprise Linux 7, and restore a backup of the 3.6 Manager database on the new Manager.

Important

The upgrade utility builds a new Manager based on a template. Manual changes or custom configuration to the original Manager such as custom users, SSH keys, and monitoring must be reapplied manually on the new Manager.

Prerequisites

  • All hosts in the environment must be running Red Hat Enterprise Linux 7.
  • All data centers and clusters in the environment must have a compatibility version of 3.6.
  • The /var/tmp directory must have at least 5 GB of free space to extract the appliance files. If it does not, you can specify a different directory or mount alternate storage that does have the required space. The VDSM user and KVM group must have read, write, and execute permissions on the directory.
  • The self-hosted engine storage domain must have additional free space for the new appliance being deployed (50 GB by default). To increase the storage on iSCSI or Fibre Channel storage, you must manually extend the LUN size on the storage and then extend the storage domain using the Manager. See Increasing iSCSI or FCP Storage in the Red Hat Enterprise Virtualization 3.6 Administration Guide for more information about resizing a LUN.

Procedure

  1. On the host that is currently set as SPM and contains the Manager virtual machine, enable the required repository for Red Hat Virtualization 4.0:

    # subscription-manager repos --enable=rhel-7-server-rhv-4-mgmt-agent-rpms
  2. Migrate all virtual machines except the Manager virtual machine to alternate hosts.
  3. On the host, update the Manager virtual machine packages:

    # yum update ovirt-hosted-engine-setup rhevm-appliance

    If the rhevm-appliance package is missing, install it manually before updating ovirt-hosted-engine-setup.

    # yum install rhevm-appliance
    # yum update ovirt-hosted-engine-setup
  4. Run the upgrade utility to upgrade the Manager virtual machine. If not already installed, install the screen package, which is available in the standard Red Hat Enterprise Linux repository:

    # yum install screen
    # screen
    # hosted-engine --upgrade-appliance

    You will be prompted to select the appliance if more than one is detected, and to create a backup of the Manager database and provide its full location.

If anything went wrong during the upgrade, power off the Manager by using the hosted-engine --vm-poweroff command, then roll back the upgrade by running hosted-engine --rollback-upgrade.

The backup created during the upgrade is not automatically deleted. You can manually delete it after confirming the upgrade was successful. The backup disks are labeled with hosted-engine-backup-*.

8.1.4. Upgrading the Manager from 4.0 to 4.1

Upgrade the Red Hat Virtualization Manager from 4.0 to 4.1.

Important

If the upgrade fails, the engine-setup command will attempt to roll your Red Hat Virtualization Manager installation back to its previous state. For this reason, the previous version’s repositories must not be removed until after the upgrade is complete. If the upgrade fails, detailed instructions display that explain how to restore your installation.

Procedure

  1. Enable the Red Hat Virtualization 4.1 repositories:

    # subscription-manager repos \
        --enable=rhel-7-server-rhv-4.1-rpms \
        --enable=rhel-7-server-rhv-4-tools-rpms \
        --enable=jb-eap-7.1-for-rhel-7-server-rpms

    All other repositories remain the same across Red Hat Virtualization releases.

  2. Update the setup packages:

    # yum update ovirt\*setup\*
  3. Run engine-setup and follow the prompts to upgrade the Red Hat Virtualization Manager:

    # engine-setup
  4. Disable the Red Hat Virtualization 4.0 repositories to ensure the system does not use any 4.0 packages:

    # subscription-manager repos \
        --disable=rhel-7-server-rhv-4.0-rpms \
        --disable=jb-eap-7-for-rhel-7-server-rpms \
        --disable=jb-eap-7.0-for-rhel-7-server-rpms
  5. Update the base operating system:

    # yum update
  6. Reboot the machine.

8.1.5. Upgrading the Manager from 4.1 to 4.2

Upgrade the Red Hat Virtualization Manager from 4.1 to 4.2.

Important

If the upgrade fails, the engine-setup command will attempt to roll your Red Hat Virtualization Manager installation back to its previous state. For this reason, the previous version’s repositories must not be removed until after the upgrade is complete. If the upgrade fails, detailed instructions display that explain how to restore your installation.

Procedure

  1. Enable the Red Hat Virtualization 4.2 repositories:

    # subscription-manager repos \
        --enable=rhel-7-server-rhv-4.2-manager-rpms \
        --enable=rhel-7-server-rhv-4-manager-tools-rpms \
        --enable=jb-eap-7-for-rhel-7-server-rpms \
        --enable=rhel-7-server-ansible-2.9-rpms

    All other repositories remain the same across Red Hat Virtualization releases.

  2. Update the setup packages:

    # yum update ovirt\*setup\*
  3. Run engine-setup and follow the prompts to upgrade the Red Hat Virtualization Manager:

    # engine-setup
    Note

    If you changed the default trust keystore password, you might get a keystore certificate error. If so, update the configuration to reflect the correct password. For more information, see RHV 4.2 - Failed to import provider certificate into the external provider keystore

  4. Disable the Red Hat Virtualization 4.1 repositories to ensure the system does not use any 4.1 packages:

    # subscription-manager repos \
        --disable=rhel-7-server-rhv-4.1-rpms \
        --disable=rhel-7-server-rhv-4.1-manager-rpms \
        --disable=rhel-7-server-rhv-4-tools-rpms \
        --disable=jb-eap-7.0-for-rhel-7-server-rpms \
        --disable=jb-eap-7.1-for-rhel-7-server-rpms
  5. Update the base operating system:

    # yum update
  6. Reboot the machine.

8.1.6. Disabling Global Maintenance Mode

Procedure

  1. Log in to one of the self-hosted engine nodes and disable global maintenance mode:

    # hosted-engine --set-maintenance --mode=none
  2. Confirm that the environment is running:

    # hosted-engine --vm-status

You can now update the self-hosted engine nodes, and then any standard hosts. The procedure is the same for both host types.

8.1.7. Updating the Hosts

Important

Use this procedure to update Red Hat Enterprise Linux hosts or Red Hat Virtualization Hosts (RHVH).

Legacy Red Hat Enterprise Virtualization Hypervisors (RHEV-H) are not supported in Red Hat Virtualization; you must reinstall them with RHVH. See Installing Red Hat Virtualization Host in the Installation Guide. If you need to preserve local storage on the host, see Appendix B, Upgrading from RHEV-H 3.6 to RHVH 4.2 While Preserving Local Storage.

If you are not sure whether you are using RHEV-H or RHVH, type imgbase check. If the command fails, the host is RHEV-H. If it succeeds, the host is RHVH.

Use the host upgrade manager to update individual hosts directly from the Red Hat Virtualization Manager.

Note

The upgrade manager only checks hosts with a status of Up or Non-operational, but not Maintenance.

Important

On RHVH, the update only preserves modified content in the /etc and /var directories. Modified data in other paths is overwritten during an update.

Prerequisites

  • If migration is enabled at the cluster level, virtual machines are automatically migrated to another host in the cluster. Update a host when its usage is relatively low.
  • Ensure that the cluster contains more than one host before performing an update. Do not update all hosts at the same time, as one host must remain available to perform Storage Pool Manager (SPM) tasks.
  • Ensure that the cluster to which the host belongs has sufficient memory reserve for its hosts to perform maintenance. Otherwise, the virtual machine migration operation will hang and fail. You can reduce the memory usage of this operation by shutting down some or all virtual machines before updating the host.
  • You cannot migrate a virtual machine using a vGPU to a different host. Virtual machines with vGPUs installed must be shut down before updating the host.

Procedure

  1. If your Red Hat Enterprise Linux hosts are locked to version 7.3, as described in https://access.redhat.com/solutions/3194482, set them to the general RHEL 7 version before updating (to view the version number, type subscription-manager release --show):

    # subscription-manager release --set=7Server
  2. Disable your current repositories:

    # subscription-manager repos --disable='*'
  3. Ensure that the correct repositories are enabled (to view a list of currently enabled repositories, type yum repolist):

    • For Red Hat Virtualization Hosts:

      # subscription-manager repos --enable=rhel-7-server-rhvh-4-rpms
    • For Red Hat Enterprise Linux hosts:

      # subscription-manager repos \
          --enable=rhel-7-server-rpms \
          --enable=rhel-7-server-rhv-4-mgmt-agent-rpms \
          --enable=rhel-7-server-ansible-2.9-rpms
  4. In the Administration Portal, click ComputeHosts and select the host to be updated.
  5. Click InstallationCheck for Upgrade and click OK.

    Click the Events and alerts notification icon ( EventsIcon ) and expand the Events section to see the result.

  6. If an update is available, click InstallationUpgrade.
  7. Click OK to update the host. Running virtual machines will be migrated according to their migration policy. If migration is disabled for any virtual machines, you will be prompted to shut them down.

    The details of the host are updated in ComputeHosts and the status transitions through these stages:

    • Maintenance
    • Installing
    • Reboot
    • Up

      If any virtual machines were migrated off the host, they are now migrated back.

      Note

      If the update fails, the host’s status changes to Install Failed. From Install Failed you can click InstallationUpgrade again.

Repeat this procedure for each host in the Red Hat Virtualization environment.

8.1.8. Changing the Cluster Compatibility Version

Red Hat Virtualization clusters have a compatibility version. The cluster compatibility version indicates the features of Red Hat Virtualization supported by all of the hosts in the cluster. The cluster compatibility is set according to the version of the least capable host operating system in the cluster.

Important

To change the cluster compatibility version, you must have first updated all the hosts in your cluster to a level that supports your desired compatibility level. Check if there is an icon next to the host indicating an update is available.

Procedure

  1. Click ComputeClusters and select the cluster to change.
  2. Click Edit.
  3. Change the Compatibility Version to the desired value.
  4. Click OK to open the Change Cluster Compatibility Version confirmation window.
  5. Click OK to confirm.
Important

An error message may warn that some virtual machines and templates are incorrectly configured. To fix this error, edit each virtual machine manually. The Edit Virtual Machine window provides additional validations and warnings that show what to correct. Sometimes the issue is automatically corrected and the virtual machine’s configuration just needs to be saved again. After editing each virtual machine, you will be able to change the cluster compatibility version.

After you update the cluster’s compatibility version, you must update the cluster compatibility version of all running or suspended virtual machines by restarting them from within the Manager, or using the REST API, instead of within the guest operating system. Virtual machines will continue to run in the previous cluster compatibility level until they are restarted. Those virtual machines that require a restart are marked with the pending changes icon ( pendingchanges ). You cannot change the cluster compatibility version of a virtual machine snapshot that is in preview; you must first commit or undo the preview.

The self-hosted engine virtual machine does not need to be restarted.

Once you have updated the compatibility version of all clusters in a data center, you can then change the compatibility version of the data center itself.

8.1.9. Changing the Data Center Compatibility Version

Red Hat Virtualization data centers have a compatibility version. The compatibility version indicates the version of Red Hat Virtualization that the data center is intended to be compatible with. All clusters in the data center must support the desired compatibility level.

Important

To change the data center compatibility version, you must have first updated all the clusters in your data center to a level that supports your desired compatibility level.

Procedure

  1. Click ComputeData Centers and select the data center to change.
  2. Click Edit.
  3. Change the Compatibility Version to the desired value.
  4. Click OK to open the Change Data Center Compatibility Version confirmation window.
  5. Click OK to confirm.

8.1.10. Replacing SHA-1 Certificates with SHA-256 Certificates

Red Hat Virtualization 4.2 uses SHA-256 signatures, which provide a more secure way to sign SSL certificates than SHA-1. Newly installed 4.2 systems do not require any special steps to enable Red Hat Virtualization’s public key infrastructure (PKI) to use SHA-256 signatures. However, for upgraded systems one of the following is recommended:

Preventing Warning Messages from Appearing in the Browser
  1. Log in to the Manager machine as the root user.
  2. Check whether /etc/pki/ovirt-engine/openssl.conf includes the line default_md = sha256:

    # cat /etc/pki/ovirt-engine/openssl.conf

    If it still includes default_md = sha1, back up the existing configuration and change the default to sha256:

    # cp -p /etc/pki/ovirt-engine/openssl.conf /etc/pki/ovirt-engine/openssl.conf."$(date +"%Y%m%d%H%M%S")"
    # sed -i 's/^default_md = sha1/default_md = sha256/' /etc/pki/ovirt-engine/openssl.conf
  3. Define the certificate that should be re-signed:

    # names="apache"
  4. Log in to one of the self-hosted engine nodes and enable global maintenance:

    # hosted-engine --set-maintenance --mode=global
  5. On the Manager, re-sign the Apache certificate:

    for name in $names; do
        subject="$(
            openssl \
                x509 \
                -in /etc/pki/ovirt-engine/certs/"${name}".cer \
                -noout \
                -subject \
            | sed \
                's;subject= \(.*\);\1;' \
        )"
       /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \
            --name="${name}" \
            --password=mypass \
            --subject="${subject}" \
            --keep-key
    done
  6. Restart the httpd service:

    # systemctl restart httpd
  7. Log in to one of the self-hosted engine nodes and disable global maintenance:

    # hosted-engine --set-maintenance --mode=none
  8. Connect to the Administration Portal to confirm that the warning no longer appears.
  9. If you previously imported a CA or https certificate into the browser, find the certificate(s), remove them from the browser, and reimport the new CA certificate. Install the certificate authority according to the instructions provided by your browser. To get the certificate authority’s certificate, navigate to http://your-manager-fqdn/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA, replacing your-manager-fqdn with the fully qualified domain name (FQDN).
Replacing All Signed Certificates with SHA-256
  1. Log in to the Manager machine as the root user.
  2. Check whether /etc/pki/ovirt-engine/openssl.conf includes the line default_md = sha256:

    # cat /etc/pki/ovirt-engine/openssl.conf

    If it still includes default_md = sha1, back up the existing configuration and change the default to sha256:

    # cp -p /etc/pki/ovirt-engine/openssl.conf /etc/pki/ovirt-engine/openssl.conf."$(date +"%Y%m%d%H%M%S")"
    # sed -i 's/^default_md = sha1/default_md = sha256/' /etc/pki/ovirt-engine/openssl.conf
  3. Re-sign the CA certificate by backing it up and creating a new certificate in ca.pem.new:

    # cp -p /etc/pki/ovirt-engine/private/ca.pem /etc/pki/ovirt-engine/private/ca.pem."$(date +"%Y%m%d%H%M%S")"
    # openssl x509 -signkey /etc/pki/ovirt-engine/private/ca.pem -in /etc/pki/ovirt-engine/ca.pem -out /etc/pki/ovirt-engine/ca.pem.new -days 3650 -sha256
  4. Replace the existing certificate with the new certificate:

    # mv /etc/pki/ovirt-engine/ca.pem.new /etc/pki/ovirt-engine/ca.pem
  5. Define the certificates that should be re-signed:

    # names="engine apache websocket-proxy jboss imageio-proxy"

    If you replaced the Red Hat Virtualization Manager SSL Certificate after the upgrade, run the following instead:

    # names="engine websocket-proxy jboss imageio-proxy"

    For more details see Replacing the Red Hat Virtualization Manager SSL Certificate in the Administration Guide.

  6. Log in to one of the self-hosted engine nodes and enable global maintenance:

    # hosted-engine --set-maintenance --mode=global
  7. On the Manager, re-sign the certificates:

    for name in $names; do
       subject="$(
            openssl \
                x509 \
                -in /etc/pki/ovirt-engine/certs/"${name}".cer \
                -noout \
                -subject \
            | sed \
                's;subject= \(.*\);\1;' \
            )"
         /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \
                --name="${name}" \
                --password=mypass \
                --subject="${subject}" \
                --keep-key
    done
  8. Restart the following services:

    # systemctl restart httpd
    # systemctl restart ovirt-engine
    # systemctl restart ovirt-websocket-proxy
    # systemctl restart ovirt-imageio-proxy
  9. Log in to one of the self-hosted engine nodes and disable global maintenance:

    # hosted-engine --set-maintenance --mode=none
  10. Connect to the Administration Portal to confirm that the warning no longer appears.
  11. If you previously imported a CA or https certificate into the browser, find the certificate(s), remove them from the browser, and reimport the new CA certificate. Install the certificate authority according to the instructions provided by your browser. To get the certificate authority’s certificate, navigate to http://your-manager-fqdn/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA, replacing your-manager-fqdn with the fully qualified domain name (FQDN).
  12. Enroll the certificates on the hosts. Repeat the following procedure for each host.

    1. In the Administration Portal, click ComputeHosts.
    2. Select the host and click ManagementMaintenance.
    3. Once the host is in maintenance mode, click InstallationEnroll Certificate.
    4. Click ManagementActivate.