3.2. Upgrading a Remote Database Environment from Red Hat Virtualization 4.2 to 4.3

Upgrading your environment from 4.2 to 4.3 involves the following steps:

3.2.1. Prerequisites

  • Plan for any necessary virtual machine downtime. After you update the clusters' compatibility versions during the upgrade, a new hardware configuration is automatically applied to each virtual machine once it reboots. You must reboot any running or suspended virtual machines as soon as possible to apply the configuration changes.
  • Ensure your environment meets the requirements for Red Hat Virtualization 4.4. For a complete list of prerequisites, see the Planning and Prerequisites Guide.
  • When upgrading Red Hat Virtualization Manager, it is recommended that you use one of the existing hosts. If you decide to use a new host, you must assign a unique name to the new host and then add it to the existing cluster before you begin the upgrade procedure.

3.2.2. Analyzing the Environment

It is recommended to run the Log Collection Analysis tool and the Image Discrepancies tool prior to performing updates and for troubleshooting. These tools analyze your environment for known issues that might prevent you from performing an update, and provide recommendations to resolve them.

3.2.3. Log Collection Analysis tool

Run the Log Collection Analysis tool prior to performing updates and for troubleshooting. The tool analyzes your environment for known issues that might prevent you from performing an update, and provides recommendations to resolve them. The tool gathers detailed information about your system and presents it as an HTML file.

Prerequisites

  • Ensure the Manager has the correct repositories enabled. For the list of required repositories, see Enabling the Red Hat Virtualization Manager Repositories for Red Hat Virtualization 4.2.

    Updates to the Red Hat Virtualization Manager are released through the Content Delivery Network.

Procedure

  1. Install the Log Collection Analysis tool on the Manager machine:

    # yum install rhv-log-collector-analyzer
  2. Run the tool:

    # rhv-log-collector-analyzer --live

    A detailed report is displayed.

    By default, the report is saved to a file called analyzer_report.html.

    To save the file to a specific location, use the --html flag and specify the location:

    # rhv-log-collector-analyzer --live --html=/directory/filename.html
  3. You can use the ELinks text mode web browser to read the analyzer reports within the terminal. To install the ELinks browser:

    # yum install -y elinks
  4. Launch ELinks and open analyzer_report.html.

    # elinks /home/user1/analyzer_report.html

    To navigate the report, use the following commands in ELinks:

    • Insert to scroll up
    • Delete to scroll down
    • PageUp to page up
    • PageDown to page down
    • Left Bracket to scroll left
    • Right Bracket to scroll right

3.2.3.1. Monitoring snapshot health with the image discrepancies tool

The RHV Image Discrepancies tool analyzes image data in the Storage Domain and RHV Database. It alerts you if it finds discrepancies in volumes and volume attributes, but does not fix those discrepancies. Use this tool in a variety of scenarios, such as:

  • Before upgrading versions, to avoid carrying over broken volumes or chains to the new version.
  • Following a failed storage operation, to detect volumes or attributes in a bad state.
  • After restoring the RHV database or storage from backup.
  • Periodically, to detect potential problems before they worsen.
  • To analyze a snapshot- or live storage migration-related issues, and to verify system health after fixing these types of problems.

Prerequisites

  • Required Versions: this tool was introduced in RHV version 4.3.8 with rhv-log-collector-analyzer-0.2.15-0.el7ev.
  • Because data collection runs simultaneously at different places and is not atomic, stop all activity in the environment that can modify the storage domains. That is, do not create or remove snapshots, edit, move, create, or remove disks. Otherwise, false detection of inconsistencies may occur. Virtual Machines can remain running normally during the process.

Procedure

  1. To run the tool, enter the following command on the RHV Manager:

    # rhv-image-discrepancies
  2. If the tool finds discrepancies, rerun it to confirm the results, especially if there is a chance some operations were performed while the tool was running.
Note

This tool includes any Export and ISO storage domains and may report discrepancies for them. If so, these can be ignored, as these storage domains do not have entries for images in the RHV database.

Understanding the results

The tool reports the following:

  • If there are volumes that appear on the storage but are not in the database, or appear in the database but are not on the storage.
  • If some volume attributes differ between the storage and the database.

Sample output:

 Checking storage domain c277ad93-0973-43d9-a0ca-22199bc8e801
    Looking for missing images...
    No missing images found
    Checking discrepancies between SD/DB attributes...
    image ef325650-4b39-43cf-9e00-62b9f7659020 has a different attribute capacity on storage(2696984576) and on DB(2696986624)
    image 852613ce-79ee-4adc-a56a-ea650dcb4cfa has a different attribute capacity on storage(5424252928) and on DB(5424254976)

 Checking storage domain c64637b4-f0e8-408c-b8af-6a52946113e2
    Looking for missing images...
    No missing images found
    Checking discrepancies between SD/DB attributes...
    No discrepancies found

You can now update the Manager to the latest version of 4.2.

3.2.4. Updating the Red Hat Virtualization Manager

Prerequisites

  • Ensure the Manager has the correct repositories enabled. For the list of required repositories, see Enabling the Red Hat Virtualization Manager Repositories for Red Hat Virtualization 4.2.

    Updates to the Red Hat Virtualization Manager are released through the Content Delivery Network.

Procedure

  1. On the Manager machine, check if updated packages are available:

    # engine-upgrade-check
  2. Update the setup packages:

    # yum update ovirt\*setup\* rh\*vm-setup-plugins
  3. Update the Red Hat Virtualization Manager with the engine-setup script. The engine-setup script prompts you with some configuration questions, then stops the ovirt-engine service, downloads and installs the updated packages, backs up and updates the database, performs post-installation configuration, and starts the ovirt-engine service.

    # engine-setup

    When the script completes successfully, the following message appears:

    Execution of setup completed successfully
    Note

    The engine-setup script is also used during the Red Hat Virtualization Manager installation process, and it stores the configuration values supplied. During an update, the stored values are displayed when previewing the configuration, and might not be up to date if engine-config was used to update configuration after installation. For example, if engine-config was used to update SANWipeAfterDelete to true after installation, engine-setup will output "Default SAN wipe after delete: False" in the configuration preview. However, the updated values will not be overwritten by engine-setup.

    Important

    The update process might take some time. Do not stop the process before it completes.

  4. Update the base operating system and any optional packages installed on the Manager:

    # yum update --nobest
    Important

    If you encounter a required Ansible package conflict during the update, see Cannot perform yum update on my RHV manager (ansible conflict).

    Important

    If any kernel packages were updated, reboot the machine to complete the update.

3.2.5. Upgrading remote databases from PostgreSQL 9.5 to 10

Red Hat Virtualization 4.3 uses PostgreSQL 10 instead of PostgreSQL 9.5. If your databases are installed locally, the upgrade script automatically upgrades them from version 9.5 to 10. However, if either of your databases (Manager or Data Warehouse) is installed on a separate machine, you must perform the following procedure on each remote database before upgrading the Manager.

  1. Stop the service running on the machine:

    • When upgrading the Manager database, stop the ovirt-engine service on the Manager machine:

      # systemctl stop ovirt-engine
    • When upgrading the Data Warehouse database, stop the ovirt-engine-dwhd service on the Data Warehouse machine:

      # systemctl stop ovirt-engine-dwhd
  2. Enable the required repository to receive the PostgreSQL 10 package:

    Enable either the Red Hat Virtualization Manager repository:

    # subscription-manager repos --enable=rhel-7-server-rhv-4.3-manager-rpms

    or the SCL repository:

    # subscription-manager repos --enable rhel-server-rhscl-7-rpms
  3. Install the PostgreSQL 10 packages:

    # yum install rh-postgresql10 rh-postgresql10-postgresql-contrib
  4. Stop and disable the PostgreSQL 9.5 service:

    # systemctl stop rh-postgresql95-postgresql
    # systemctl disable rh-postgresql95-postgresql
  5. Upgrade the PostgreSQL 9.5 database to PostgreSQL 10:

    # scl enable rh-postgresql10 -- postgresql-setup --upgrade-from=rh-postgresql95-postgresql --upgrade
  6. Start and enable the rh-postgresql10-postgresql.service and check that it is running:

    # systemctl start rh-postgresql10-postgresql.service
    # systemctl enable rh-postgresql10-postgresql.service
    # systemctl status rh-postgresql10-postgresql.service

    Ensure that you see output similar to the following:

    rh-postgresql10-postgresql.service - PostgreSQL database server
       Loaded: loaded (/usr/lib/systemd/system/rh-postgresql10-postgresql.service;
    enabled; vendor preset: disabled)
       Active: active (running) since ...
  7. Copy the pg_hba.conf client configuration file from the PostgreSQL 9.5 environment to the PostgreSQL 10 environment:

    # cp -p /var/opt/rh/rh-postgresql95/lib/pgsql/data/pg_hba.conf  /var/opt/rh/rh-postgresql10/lib/pgsql/data/pg_hba.conf
  8. Update the following parameters in /var/opt/rh/rh-postgresql10/lib/pgsql/data/postgresql.conf:

    listen_addresses='*'
    autovacuum_vacuum_scale_factor=0.01
    autovacuum_analyze_scale_factor=0.075
    autovacuum_max_workers=6
    maintenance_work_mem=65536
    max_connections=150
    work_mem = 8192
  9. Restart the PostgreSQL 10 service to apply the configuration changes:

    # systemctl restart rh-postgresql10-postgresql.service

You can now upgrade the Manager to 4.3.

3.2.6. Upgrading the Red Hat Virtualization Manager from 4.2 to 4.3

Follow these same steps when upgrading any of the following:

  • the Red Hat Virtualization Manager
  • a remote machine with the Data Warehouse service

You need to be logged into the machine that you are upgrading.

Important

If the upgrade fails, the engine-setup command attempts to restore your Red Hat Virtualization Manager installation to its previous state. For this reason, do not remove the previous version’s repositories until after the upgrade is complete. If the upgrade fails, the engine-setup script explains how to restore your installation.

Procedure

  1. Enable the Red Hat Virtualization 4.3 repositories:

    # subscription-manager repos \
        --enable=rhel-7-server-rhv-4.3-manager-rpms \
        --enable=jb-eap-7.2-for-rhel-7-server-rpms

    All other repositories remain the same across Red Hat Virtualization releases.

  2. Update the setup packages:

    # yum update ovirt\*setup\* rh\*vm-setup-plugins
  3. Run engine-setup and follow the prompts to upgrade the Red Hat Virtualization Manager, the remote database or remote service:

    # engine-setup
    Note

    During the upgrade process for the Manager, the engine-setup script might prompt you to disconnect the remote Data Warehouse database. You must disconnect it to continue the setup.

    When the script completes successfully, the following message appears:

    Execution of setup completed successfully
  4. Disable the Red Hat Virtualization 4.2 repositories to ensure the system does not use any 4.2 packages:

    # subscription-manager repos \
        --disable=rhel-7-server-rhv-4.2-manager-rpms \
        --disable=jb-eap-7-for-rhel-7-server-rpms
  5. Update the base operating system:

    # yum update
    Important

    If you encounter a required Ansible package conflict during the update, see Cannot perform yum update on my RHV manager (ansible conflict).

    Important

    If any kernel packages were updated, reboot the machine to complete the upgrade.

The Manager is now upgraded to version 4.3.

3.2.6.1. Completing the remote Data Warehouse database upgrade

Complete these additional steps when upgrading a remote Data Warehouse database from PostgreSQL 9.5 to 10.

Procedure

  1. The ovirt-engine-dwhd service is now running on the Manager machine. If the ovirt-engine-dwhd service is on a remote machine, stop and disable the ovirt-engine-dwhd service on the Manager machine, and remove the configuration files that engine-setup created:

    # systemctl stop ovirt-engine-dwhd
    # systemctl disable ovirt-engine-dwhd
    # rm -f /etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/*
  2. Repeat the steps in Upgrading the Manager to 4.3 on the machine hosting the ovirt-engine-dwhd service.

You can now update the hosts.

3.2.7. Updating All Hosts in a Cluster

You can update all hosts in a cluster instead of updating hosts individually. This is particularly useful during upgrades to new versions of Red Hat Virtualization. See oVirt Cluster Upgrade for more information about the Ansible role used to automate the updates.

Update one cluster at a time.

Limitations

  • On RHVH, the update only preserves modified content in the /etc and /var directories. Modified data in other paths is overwritten during an update.
  • If the cluster has migration enabled, virtual machines are automatically migrated to another host in the cluster.
  • In a self-hosted engine environment, the Manager virtual machine can only migrate between self-hosted engine nodes in the same cluster. It cannot migrate to standard hosts.
  • The cluster must have sufficient memory reserved for its hosts to perform maintenance. Otherwise, virtual machine migrations will hang and fail. You can reduce the memory usage of host updates by shutting down some or all virtual machines before updating hosts.
  • You cannot migrate a pinned virtual machine (such as a virtual machine using a vGPU) to another host. Pinned virtual machines are shut down during the update, unless you choose to skip that host instead.

Procedure

  1. In the Administration Portal, click ComputeClusters and select the cluster. The Upgrade status column shows if an upgrade is available for any hosts in the cluster.
  2. Click Upgrade.
  3. Select the hosts to update, then click Next.
  4. Configure the options:

    • Stop Pinned VMs shuts down any virtual machines that are pinned to hosts in the cluster, and is selected by default. You can clear this check box to skip updating those hosts so that the pinned virtual machines stay running, such as when a pinned virtual machine is running important services or processes and you do not want it to shut down at an unknown time during the update.
    • Upgrade Timeout (Minutes) sets the time to wait for an individual host to be updated before the cluster upgrade fails with a timeout. The default is 60. You can increase it for large clusters where 60 minutes might not be enough, or reduce it for small clusters where the hosts update quickly.
    • Check Upgrade checks each host for available updates before running the upgrade process. It is not selected by default, but you can select it if you need to ensure that recent updates are included, such as when you have configured the Manager to check for host updates less frequently than the default.
    • Reboot After Upgrade reboots each host after it is updated, and is selected by default. You can clear this check box to speed up the process if you are sure that there are no pending updates that require a host reboot.
    • Use Maintenance Policy sets the cluster’s scheduling policy to cluster_maintenance during the update. It is selected by default, so activity is limited and virtual machines cannot start unless they are highly available. You can clear this check box if you have a custom scheduling policy that you want to keep using during the update, but this could have unknown consequences. Ensure your custom policy is compatible with cluster upgrade activity before disabling this option.
  5. Click Next.
  6. Review the summary of the hosts and virtual machines that will be affected.
  7. Click Upgrade.

You can track the progress of host updates:

  • in the ComputeClusters view, the Upgrade Status column shows Upgrade in progress.
  • in the ComputeHosts view
  • in the Events section of the Notification Drawer ( EventsIcon ).

You can track the progress of individual virtual machine migrations in the Status column of the ComputeVirtual Machines view. In large environments, you may need to filter the results to show a particular group of virtual machines.

3.2.8. Changing the Cluster Compatibility Version

Red Hat Virtualization clusters have a compatibility version. The cluster compatibility version indicates the features of Red Hat Virtualization supported by all of the hosts in the cluster. The cluster compatibility is set according to the version of the least capable host operating system in the cluster.

Prerequisites

  • To change the cluster compatibility level, you must first update all the hosts in your cluster to a level that supports your desired compatibility level. Check if there is an icon next to the host indicating an update is available.

Limitations

  • Virtio NICs are enumerated as a different device after upgrading the cluster compatibility level to 4.6. Therefore, the NICs might need to be reconfigured. Red Hat recommends that you test the virtual machines before you upgrade the cluster by setting the cluster compatibility level to 4.6 on the virtual machine and verifying the network connection.

    If the network connection for the virtual machine fails, configure the virtual machine with a custom emulated machine that matches the current emulated machine, for example pc-q35-rhel8.3.0 for 4.5 compatibility version, before upgrading the cluster.

Procedure

  1. In the Administration Portal, click ComputeClusters.
  2. Select the cluster to change and click Edit.
  3. On the General tab, change the Compatibility Version to the desired value.
  4. Click OK. The Change Cluster Compatibility Version confirmation dialog opens.
  5. Click OK to confirm.
Important

An error message might warn that some virtual machines and templates are incorrectly configured. To fix this error, edit each virtual machine manually. The Edit Virtual Machine window provides additional validations and warnings that show what to correct. Sometimes the issue is automatically corrected and the virtual machine’s configuration just needs to be saved again. After editing each virtual machine, you will be able to change the cluster compatibility version.

3.2.9. Changing Virtual Machine Cluster Compatibility

After updating a cluster’s compatibility version, you must update the cluster compatibility version of all running or suspended virtual machines by rebooting them from the Administration Portal, or using the REST API, or from within the guest operating system. Virtual machines that require a reboot are marked with the pending changes icon ( pendingchanges ).

Although you can wait to reboot the virtual machines at a convenient time, rebooting immediately is highly recommended so that the virtual machines use the latest configuration. Any virtual machine that has not been rebooted runs with the previous configuration, and subsequent configuration changes made to the virtual machine might overwrite its pending cluster compatibility changes.

Procedure

  1. In the Administration Portal, click ComputeVirtual Machines.
  2. Check which virtual machines require a reboot. In the Vms: search bar, enter the following query:

    next_run_config_exists=True

    The search results show all virtual machines with pending changes.

  3. Select each virtual machine and click Restart. Alternatively, if necessary you can reboot a virtual machine from within the virtual machine itself.

When the virtual machine starts, the new compatibility version is automatically applied.

Note

You cannot change the cluster compatibility version of a virtual machine snapshot that is in preview. You must first commit or undo the preview.

3.2.10. Changing the Data Center Compatibility Version

Red Hat Virtualization data centers have a compatibility version. The compatibility version indicates the version of Red Hat Virtualization with which the data center is intended to be compatible. All clusters in the data center must support the desired compatibility level.

Prerequisites

  • To change the data center compatibility level, you must first update the compatibility version of all clusters and virtual machines in the data center.

Procedure

  1. In the Administration Portal, click ComputeData Centers.
  2. Select the data center to change and click Edit.
  3. Change the Compatibility Version to the desired value.
  4. Click OK. The Change Data Center Compatibility Version confirmation dialog opens.
  5. Click OK to confirm.

If you previously upgraded to 4.2 without replacing SHA-1 certificates with SHA-256 certificates, you must do so now.

3.2.11. Replacing SHA-1 Certificates with SHA-256 Certificates

Red Hat Virtualization 4.4 uses SHA-256 signatures, which provide a more secure way to sign SSL certificates than SHA-1. Newly installed systems do not require any special steps to enable Red Hat Virtualization’s public key infrastructure (PKI) to use SHA-256 signatures.

Warning

Do NOT let certificates expire. If they expire, the environment becomes non-responsive and recovery is an error prone and time consuming process. For information on renewing certificates, see Renewing certificates before they expire in the Administration Guide.

Preventing Warning Messages from Appearing in the Browser
  1. Log in to the Manager machine as the root user.
  2. Check whether /etc/pki/ovirt-engine/openssl.conf includes the line default_md = sha256:

    # cat /etc/pki/ovirt-engine/openssl.conf

    If it still includes default_md = sha1, back up the existing configuration and change the default to sha256:

    # cp -p /etc/pki/ovirt-engine/openssl.conf /etc/pki/ovirt-engine/openssl.conf."$(date +"%Y%m%d%H%M%S")"
    # sed -i 's/^default_md = sha1/default_md = sha256/' /etc/pki/ovirt-engine/openssl.conf
  3. Define the certificate that should be re-signed:

    # names="apache"
  4. On the Manager, save a backup of the /etc/ovirt-engine/engine.conf.d and /etc/pki/ovirt-engine directories, and re-sign the certificates:

    # . /etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf
    # for name in $names; do
        subject="$(
            openssl \
                x509 \
                -in /etc/pki/ovirt-engine/certs/"${name}".cer \
                -noout \
                -subject \
                -nameopt compat \
            | sed \
                's;subject=\(.*\);\1;' \
        )"
       /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \
            --name="${name}" \
            --password=mypass \ <1>
            --subject="${subject}" \
            --san=DNS:"${ENGINE_FQDN}" \
            --keep-key
    done
    Do not change this the password value.
  5. Restart the httpd service:

    # systemctl restart httpd
  6. Connect to the Administration Portal to confirm that the warning no longer appears.
  7. If you previously imported a CA or https certificate into the browser, find the certificate(s), remove them from the browser, and reimport the new CA certificate. Install the certificate authority according to the instructions provided by your browser. To get the certificate authority’s certificate, navigate to http://your-manager-fqdn/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA, replacing your-manager-fqdn with the fully qualified domain name (FQDN).
Replacing All Signed Certificates with SHA-256
  1. Log in to the Manager machine as the root user.
  2. Check whether /etc/pki/ovirt-engine/openssl.conf includes the line default_md = sha256:

    # cat /etc/pki/ovirt-engine/openssl.conf

    If it still includes default_md = sha1, back up the existing configuration and change the default to sha256:

    # cp -p /etc/pki/ovirt-engine/openssl.conf /etc/pki/ovirt-engine/openssl.conf."$(date +"%Y%m%d%H%M%S")"
    # sed -i 's/^default_md = sha1/default_md = sha256/' /etc/pki/ovirt-engine/openssl.conf
  3. Re-sign the CA certificate by backing it up and creating a new certificate in ca.pem.new:

    # cp -p /etc/pki/ovirt-engine/private/ca.pem /etc/pki/ovirt-engine/private/ca.pem."$(date +"%Y%m%d%H%M%S")"
    # openssl x509 -signkey /etc/pki/ovirt-engine/private/ca.pem -in /etc/pki/ovirt-engine/ca.pem -out /etc/pki/ovirt-engine/ca.pem.new -days 3650 -sha256
  4. Replace the existing certificate with the new certificate:

    # mv /etc/pki/ovirt-engine/ca.pem.new /etc/pki/ovirt-engine/ca.pem
  5. Define the certificates that should be re-signed:

    # names="engine apache websocket-proxy jboss imageio-proxy"

    If you replaced the Red Hat Virtualization Manager SSL Certificate after the upgrade, run the following instead:

    # names="engine websocket-proxy jboss imageio-proxy"

    For more details see Replacing the Red Hat Virtualization Manager CA Certificate in the Administration Guide.

  6. On the Manager, save a backup of the /etc/ovirt-engine/engine.conf.d and /etc/pki/ovirt-engine directories, and re-sign the certificates:

    # . /etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf
    # for name in $names; do
        subject="$(
            openssl \
                x509 \
                -in /etc/pki/ovirt-engine/certs/"${name}".cer \
                -noout \
                -subject \
                -nameopt compat \
            | sed \
                's;subject=\(.*\);\1;' \
        )"
       /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \
            --name="${name}" \
            --password=mypass \ <1>
            --subject="${subject}" \
            --san=DNS:"${ENGINE_FQDN}" \
            --keep-key
    done
    Do not change this the password value.
  7. Restart the following services:

    # systemctl restart httpd
    # systemctl restart ovirt-engine
    # systemctl restart ovirt-websocket-proxy
    # systemctl restart ovirt-imageio
  8. Connect to the Administration Portal to confirm that the warning no longer appears.
  9. If you previously imported a CA or https certificate into the browser, find the certificate(s), remove them from the browser, and reimport the new CA certificate. Install the certificate authority according to the instructions provided by your browser. To get the certificate authority’s certificate, navigate to http://your-manager-fqdn/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA, replacing your-manager-fqdn with the fully qualified domain name (FQDN).
  10. Enroll the certificates on the hosts. Repeat the following procedure for each host.

    1. In the Administration Portal, click ComputeHosts.
    2. Select the host and click ManagementMaintenance and OK.
    3. Once the host is in maintenance mode, click InstallationEnroll Certificate.
    4. Click ManagementActivate.