6.3. Upgrading a RHEL or RHEV-H 6 Self-Hosted Engine Environment

This section describes how to upgrade a Red Hat Enterprise Virtualization 3.5 self-hosted engine environment, running on Red Hat Enterprise Linux (RHEL) 6 or Red Hat Enterprise Virtualization Hypervisor (RHEV-H) 6 hosts, to Red Hat Enterprise Virtualization 3.6, running on Next Generation Red Hat Virtualization Host (RHVH) or RHEL 7 hosts.
The upgrade procedure comprises the following key steps:
  1. Upgrade the Manager
  2. Upgrade the hosts
  3. Perform post-upgrade tasks
This procedure describes how to upgrade the Manager from 3.5 to 3.6. This procedure assumes that the system on which the Manager is installed is subscribed to the entitlements for receiving Red Hat Enterprise Virtualization 3.5 packages at the start of the procedure.

Important

If the upgrade fails, the engine-setup command will attempt to roll back the Manager installation to its previous state. For this reason, the repositories required by Red Hat Enterprise Virtualization 3.5 must not be removed until after the upgrade is complete. If the upgrade fails, detailed instructions display that explain how to restore your installation.

Procedure 6.9. Upgrading the Manager

  1. Enable global maintenance from one of the self-hosted engine hosts: 
    # hosted-engine --set-maintenance --mode=global
  2. On the Manager virtual machine, subscribe to the required entitlements for receiving the Manager 3.6 packages: 
    # subscription-manager repos --enable=rhel-6-server-rhevm-3.6-rpms
  3. Update the rhevm setup packages: 
    # yum update rhevm\*setup\*
  4. Run the following command and follow the prompts to upgrade the Manager: 
    # engine-setup --otopi-environment=OVESETUP_DB/forceUpgradeWithHeEl6Hosts=bool:True
  5. Remove or disable the Manager 3.5 repository to ensure the system does not use any Manager 3.5 packages: 
    # subscription-manager repos --disable=rhel-6-server-rhevm-3.5-rpms
After upgrading the Manager, you can upgrade the hosts.
These procedures describe how to upgrade the hosts to Next Generation RHVH or RHEL 7, using the host-direct-upgrade tool.
host-direct-upgrade Tool

The host-direct-upgrade tool provides a simple and fast way to upgrade a Red Hat Enterprise Virtualization 3.5 self-hosted engine environment, running on RHEV-H 6 or RHEL 6, to a Red Hat Enterprise Virtualization 3.6 self-hosted engine environment, running on Next Generation RHVH or RHEL 7. The tool has two main functions:

  • Importing configuration
    The tool imports the configuration files (answers.conf, hosted-engine.conf, vm.conf, broker.conf) from the host on which the self-hosted engine was first deployed to the Manager virtual machine for later export to upgraded hosts. All files must be present on the host in order for the import to succeed. During this operation, the ovirt-engine service may be restarted to allow clusters with mixed hosts (different RHEL versions).
  • Exporting configuration and configuring hosts
    The tool configures an upgraded host with the configuration files that were imported to the Manager virtual machine. This operation can only be performed after the configuration file import operation.

This procedure describes how to upgrade the host on which the self-hosted engine was originally deployed.

Procedure 6.10. Upgrading the Self-Hosted Engine Host

  1. In the Administration Portal, click the Hosts tab, select the host on which the self-hosted engine was originally deployed, click Maintenance, and click OK. The host's virtual machines, including the Manager, will migrate to other hosts.
  2. Connect to the Manager virtual machine and run the host-direct-upgrade tool to import the configuration files: 
    # /usr/share/ovirt-engine/bin/host-direct-upgrade.sh --import-configuration-files [-h 3.5-host-address]

    Note

    If you do not specify the optional -h 3.5-host-address parameter, you will be prompted to input the 3.5 host's IP address or FQDN later.
    The host-direct-upgrade tool checks whether the self-hosted engine is configured to allow mixed RHEL versions in the same cluster. If not, the ovirt-engine service is restarted.
  3. In the Administration Portal, click the Clusters tab and select the cluster.
  4. Click Edit.
  5. In the Scheduling Policy tab, select InClusterUpgrade from the Policy drop-down list and click OK.
  6. Download Next Generation RHVH (for RHEV-M 3.6.11 Async) or RHEL 7.
  7. Install the newer operating system on the host, keeping the same IP address and host name. See Installing Red Hat Virtualization Host in the Installation Guide to install Next Generation RHVH or the Red Hat Enterprise Linux Installation Guide to install RHEL 7.
  8. Connect to the Manager virtual machine and run the host-direct-upgrade tool to export the configuration files: 
    # /usr/share/ovirt-engine/bin/host-direct-upgrade.sh --export-and-configure [-h 3.6-host-address]

    Note

    The 3.6-host-address is the IP address or FQDN of the host that you upgraded.
    This process may take 10 to 15 minutes.
  9. Check the host's status: 
    # hosted-engine --vm-status
    The Score should be 2800: 
    !! Cluster is in GLOBAL MAINTENANCE mode !!

--== Host 1 status ==--

conf_on_shared_storage             : False
Status up-to-date                  : True
Hostname                           : rhvhhe20170926h1.localdomain
Host ID                            : 1
Engine status                      : "reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"
Score                              : 2800
stopped                            : False
Local maintenance                  : False
crc32                              : ffe0a244
local_conf_timestamp               : 0
Host timestamp                     : 1375
  10. In the Administration Portal, click the Hosts tab and select the 3.5 host (still in maintenance mode).
  11. Click Remove and OK.
  12. Click New and add the upgraded host. Its status should be Up.
  13. Migrate the Manager virtual machine manually to the upgraded host. See Manually Migrating Virtual Machines in the Virtual Machine Management Guide for details.
After upgrading the Manager, check that the Manager's SPM ID matches the Host ID of each host.

Procedure 6.11. Comparing the Manager's SPM ID to the Host IDs

  1. Connect to the Manager virtual machine and run the following command to output the SPM ID: 
    # sudo -u postgres psql engine -c "SELECT vds_spm_id, vds.vds_name FROM vds"
  2. Connect to each host and run the following command to ouput the Host ID: 
    # grep "^host_id" /etc/ovirt-hosted-engine/hosted-engine.conf
  3. If a host's Host ID does not match the Manager's SPM ID:
    1. In the Administration Portal, click the Hosts tab, select the host with the mismatched Host ID, click Maintenance, and click OK.
    2. Connect to the host and correct the host_id value in the /etc/ovirt-hosted-engine/hosted-engine.conf file.
    3. Reboot the host.
    4. In the Administration Portal, click the Hosts tab and select the host.
    5. Click Activate and OK.
This procedure describes how to upgrade the remaining hosts. You must perform this procedure on each host.

Procedure 6.12. Upgrading the Remaining Hosts

  1. In the Administration Portal, click the Hosts tab, select another host, and click Maintenance.
    The host's virtual machines will migrate to other machines, preferring hosts with newer operating systems, because of the InClusterUpgrade policy. If the virtual machines do not migrate automatically, migrate them manually. See Manually Migrating Virtual Machines in the Virtual Machine Management Guide for details.
  2. When the host has no virtual machines, click Remove.
  3. Install Next Generation RHVH or RHEL 7 on the host, keeping its IP address and host name.
  4. In the Administration Portal, click the Hosts tab and click Add to add the upgraded host.

    Note

    The upgraded host is added manually, before the hosted-engine setup, in order to preserve its Host ID.
  5. Connect to the host, disable NetworkManager, and enable the network script: 
    # systemctl stop NetworkManager; systemctl disable NetworkManager; systemctl start network; systemctl enable network
  6. Install the screen utility if necessary: 
    # yum install screen
  7. Run the hosted-engine setup in a screen session: 
    # hosted-engine --deploy
    
[ INFO  ] Stage: Initializing
[ INFO  ] Generating a temporary VNC password.   
[ INFO  ] Stage: Environment setup
          Continuing will configure this host for serving as hypervisor and create a VM where you have to install the engine afterwards.
          Are you sure you want to continue? (Yes, No)[Yes]:
          Configuration files: []
          Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup.log
          Version: otopi-1.4.2 (otopi-1.4.2-1.el7ev)
[ INFO  ] Hardware supports virtualization
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup
[ INFO  ] Stage: Environment customization

          --== STORAGE CONFIGURATION ==--

          During customization use CTRL-D to abort.
          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs3, nfs4)[nfs3]: nfs4
          Please specify the full shared storage connection path to use (example: host:/path): hostname:/path
          The specified storage location already contains a data domain. Is this an additional host setup (Yes, No)[Yes]?
[ INFO  ] Installing on additional host
          Please specify the Host ID [Must be integer, default: 2]:

          --== SYSTEM CONFIGURATION ==--

[WARNING] A configuration file must be supplied to deploy Hosted Engine on an additional host.
[ INFO  ] Answer file successfully loaded

          --== NETWORK CONFIGURATION ==--

[ INFO  ] Bridge rhevm already created
[ INFO  ] Additional host deployment, firewall manager is 'iptables'
          The following CPU types are supported by this host:
                 - model_SandyBridge: Intel SandyBridge Family
                 - model_Westmere: Intel Westmere Family
                 - model_Nehalem: Intel Nehalem Family
                 - model_Penryn: Intel Penryn Family
                 - model_Conroe: Intel Conroe Family

          --== HOSTED ENGINE CONFIGURATION ==--  

          Enter the name which will be used to identify this host inside the Administrator Portal [hosted_engine_2]:
          Enter 'admin@internal' user password that will be used for accessing the Administrator Portal:
          Confirm 'admin@internal' user password:
[ INFO  ] Stage: Setup validation
          Please provide the address of this host.
          Note: The engine VM and all the other hosts should be able to correctly resolve it.
          Host address:  [rhvh.example.com]:
	  The Host ID is already known. Is this a re-deployment on an additional host that was previously set up (Yes, No)[Yes]?

          --== CONFIGURATION PREVIEW ==--

          Engine FQDN                        : enginevm.localdomain
          Bridge name                        : rhevm
          Host address                       : rhvh.example.com
          SSH daemon port                    : 22
          Firewall manager                   : iptables
          Gateway address                    : 192.168.1.1
          Host name for web application      : hosted_engine_2
          Storage Domain type                : nfs4
          Host ID                            : 2 
          Image size GB                      : 50
          GlusterFS Share Name               : hosted_engine_glusterfs
          GlusterFS Brick Provisioning       : False
          Storage connection                 : hostname:/path
          Console type                       : vnc
          Memory size MB                     : 4096
          MAC address                        : 00:16:3e:12:96:6b
          Boot type                          : disk
          Number of CPUs                     : 1 
          CPU Type                           : model_SandyBridge
[ INFO  ] Stage: Transaction setup
[ INFO  ] Stage: Misc configuration
[ INFO  ] Stage: Package installation
[ INFO  ] Stage: Misc configuration
[ INFO  ] Configuring libvirt
[ INFO  ] Configuring VDSM
[ INFO  ] Starting vdsmd
[ INFO  ] Waiting for VDSM hardware info
[ INFO  ] Configuring VM
[ INFO  ] Updating hosted-engine configuration   
[ INFO  ] Stage: Transaction commit
[ INFO  ] Stage: Closing up
[ INFO  ] Acquiring internal CA cert from the engine
[ INFO  ] The following CA certificate is going to be used, please immediately interrupt if not correct:
[ INFO  ] Issuer: C=US, O=localdomain, CN=enginevm.localdomain.34970, Subject: C=US, O=localdomain, CN=enginevm.localdomain.34970, Fingerprint (SHA-1): 46A75DA2174D03B100D1811411C0C6627CB1D821
[ INFO  ] Connecting to the Engine
[ INFO  ] Waiting for the host to become operational in the engine. This may take several minutes...

[ INFO  ] Still waiting for VDSM host to become operational...
[ INFO  ] The VDSM Host is now operational
[ ERROR ] Cannot automatically set CPU level of cluster Default: inclusterupgrade is not a member of SchedulingPolicyType. Possible values for SchedulingPolicyType are: evenly_distributed, power_saving, none, vm_evenly_distributed 
[ INFO  ] Enabling and starting HA services
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20170927190113.conf'
[ INFO  ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ INFO  ] Hosted Engine successfully set up

    Note

    You can ignore the [ ERROR ] Cannot automatically set CPU level of cluster Default message.
This procedure completes the upgrade.

Procedure 6.13. Post-Upgrade Tasks

  1. In the Administration Portal, click the Cluster tab and select the cluster.
  2. Select 3.6 from the Compatibility Version drop-down list.
  3. In the Scheduling Policy tab, select None from the Policy drop-down list and click OK.
  4. Reboot each virtual machine (except the Manager virtual machine), one by one.
  5. Connect to the Manager virtual machine and update its configuration: 
    # engine-config -s CheckMixedRhelVersions=true --cver=3.5
  6. In the Administration Portal, click the Events tab and confirm that the Manager virtual machine and its storage domain were imported successfully: 
    Sep 27, 2017 7:15:18 PM Hosted Engine VM was imported successfully
Sep 27, 2017 7:15:18 PM Starting to import Vm HostedEngine to Data Center Default, Cluster Default
Sep 27, 2017 7:15:16 PM VM HostedEngine was successfully removed.
Sep 27, 2017 7:15:00 PM Hosted Engine storage domain imported successfully
Sep 27, 2017 7:15:00 PM Storage Domain hosted_storage was attached to Data Center Default by SYSTEM
Sep 27, 2017 7:15:00 PM Storage Domain hosted_storage (Data Center Default) was activated by SYSTEM
Sep 27, 2017 7:14:57 PM The Hosted Engine Storage Domain isn't Active.
  7. In the Virtual Machines tab, select the Manager virtual machine, and click Shutdown.
  8. Connect to a self-hosted engine host and disable global maintenance mode: 
    # hosted-engine --set-maintenance --mode=none
    Wait for the ovirt-ha-agent service to restart the Manager virtual machine.
  9. Check the status of each host: 
    # hosted-engine --vm-status
    Each host should have a score of 3400: 
    
--== Host 1 status ==--

conf_on_shared_storage             : False
Status up-to-date                  : True
Hostname                           : rhvhhe20170926h1.localdomain
Host ID                            : 1
Engine status                      : "health": "good", "vm": "up", "detail": "up"
Score                              : 3400
stopped                            : False
Local maintenance                  : False
crc32                              : 6a5a17a7
local_conf_timestamp               : 7580
Host timestamp                     : 7579


--== Host 2 status ==--

conf_on_shared_storage             : False
Status up-to-date                  : True
Hostname                           : rhvhhe20170926h2.localdomain
Host ID                            : 2
Engine status                      : "reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"
Score                              : 3400
stopped                            : False
Local maintenance                  : False
crc32                              : 1c87c83d
local_conf_timestamp               : 4323
Host timestamp                     : 4323