How to upgrade RHEV 3.5 Hosted-Engine environment with RHEL 6 hosts / hypervisors to RHEV 3.6

Resolution

Prerequisites:
If using RHEV-H, get the latest Next Generation RHVH for RHEV 3.6, RHVH-3.6-20180130.0-RHVH-x86_64-dvd1.iso.

If using RHEL 7.3, make sure to lock the release version to 7.3 before upgrading to latest. (RHEL > 7.3 is not supported on RHEV 3).

# subscription-manager release --set=7.3

Note: make sure to unset the version, once the manager is on RHV 4, to be able to get latest upgrades.

Summary Steps:

• Prepare RHEV-Manager for upgrade - create a new cluster for the RHEL7 host.
• Install the latest RHVH/RHEL-H 3.6 on a new host, as described in requirements.
• Deploy it as additional Hosted-Engine host using latest RHVH/RHEL host.
  *This should trigger Hosted-Engine upgrade.
• If already on RHEL7 and 3.5 RHEV packages, upgrade it with latest RHEV 3.6 packages, and this will trigger the upgrade as well.
• Migrate Hosted-Engine to upgraded Host.
• Reinstall all other hosts with the latest version.
• Upgrade Manager to latest RHEV-M 3.6.
• Finish updating DataCenter Compatibility level to 3.6

Detailed Steps:

1. Prepare RHEV-Manager for upgrade - create a new cluster for the RHEL7 host.
Create a new temporary cluster in your environment to host the new RHEL7 hosts. You cannot mix RHEL 6 and RHEL7 hosts in one cluster in 3.5, according to this solution.

2. Install the latest RHVH 3.6 as described in prerequisites.

For the installation of RHEV-H, the ISO will need to be burned to a CD-R or written to a USB device. To write it to a USB device, check [this solution(https://access.redhat.com/articles/1168703). Some more detailed information on how to prepare and use these images can be found in the documentation. Refer to Additional Ways to Prepare the Installation Media. (it is also possible to distribute the image via PXE).

3. Deploy it as another host to HE environment
Follow the documentation. For RHEV-H, refer to Section 7.1.1.B in Adding an Additional Self-Hosted Engine Host.

Note1: Since this host is probably being reused from the current environment, make sure to put that host in maintenance first and remove it from the system via Web Admin.
Note2: If rhevm logical network is configured on top of a bond and vlan, configure the bond and the vlan via TUI before running hosted-engine --deploy.
Note3: Make sure to use the cluster created in step 1, when asked.
Note4: Can't reuse the same host ID, so if you have N hosts, input N+1.

Make sure the host was added to HE setup successfully and it can see the status of all the hosts in the system:

# hosted-engine --vm-status
# systemctl status ovirt-ha-agent
# systemctl status ovirt-ha-broker

4. Start the Hosted-Engine upgrade process

4.1. Put the new host in Maintenance Mode.

4.2. Press F2, it will drop to a shell.

4.3. Run the following commands:

# systemctl restart ovirt-ha-agent.service 
# systemctl restart ovirt-ha-broker.service

Wait until HE will finishes the upgrade procedure.
Verify that /var/log/ovirt-hosted-engine-ha/agent.log contains the message Successfully upgraded.
For detailed troubleshooting steps, check this bugzilla comment.

4.4. Activate the Host again.

5. Migrate Hosted-Engine to the upgraded host.

6. Continue reinstalling the remaining hypervisors with latest RHEV 3.6 version. No need to implement steps 1-5 above at this point, though for additional Hosted Engine hosts make sure to deploy these as Hosted Engine hosts normally.
If RHEV-H, use the latest image downloaded to RHEV-Manager at step 4. The image would be located at /usr/share/rhev-hypervisor/ on the manager.
If RHEL, reinstall them with RHEL7 OS and follow standard documentation to add RHEL host to HE environment in 3.6.

Note: if there are VMs running on those hosts, they should live migrate to newer hosts, when their current host is put in maintenance.

7. Upgrade the Manager to latest RHEV-M 3.6.
7.1. Put Hosted Engine in maintenance:

# hosted-engine --set-maintenance mode=global

7.2. Follow the documentation
7.3. Remove the hosted-engine the maintenance mode:

# hosted-engine --set-maintenance mode=none

8. Raise Cluster and Datacenter Level to 3.6
IMPORTANT: It is not required for every DC and every Cluster to be upgraded to 3.6 Compatibility Level, but the DC and Cluster that are part of the Hosted-Engine Setup must be Upgraded to 3.6 Level in order to finish the upgrade process (see next step).

9. Confirm hosted_storage (the Storage Domain of the Hosted-Engine) was Auto-Imported and is now showing up in the Storage Tab and is activated (green).
Note: if the Hosted Engine storage domain is not named hosted_storage the Auto-Import will not be triggered. If it was not Auto-Imported, please confirm in the answers file used to deploy Hosted Engine what is the name of the Hosted Storage:

OVEHOSTED_STORAGE/storageDomainName=str:my_custom_name

In this case, please set this configuration in RHV-M.

engine-config -s HostedEngineStorageDomainName=my_custom_name

Then restart the ovirt-engine service and check if the Hosted Engine my_custom_name storage Domain was Auto-Imported.

If it failed to import to Storage Domain of the Hosted-Engine with Sanlock lockspace add failure, a common cause is the import process using the wrong host id. Please see the following document further troubleshooting: How to align spm_vds_id_map with hosted-engine.conf host_id?.