RHEV-H upgrade fails with "SSH error running command <ip_addr>:'/usr/share/vdsm-reg/vdsm-upgrade'.

Issue

Failed to upgrade a hypervisor host.

An attempt to upgrade a RHEV-H 6.3 system in a RHEV 3.1 environment failed. This upgrade was invoked from the RHEV Manager using an ISO image in '/usr/share/rhev-hypervisor'.

The following sequence was reported in /var/log/ovirt-engine/engine.log on RHEV-M;

2013-01-23 13:45:05,688 INFO  [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-3-thread-49) SSH execute 10.140.66.30 '/usr/share/vdsm-reg/vdsm-upgrade'
2013-01-23 13:45:07,856 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-2) Installation of 10.140.66.30. Received message: <BSTRAP component='RHEL_INSTALL' status='OK'message='vdsm daemon stopped for upgrade process!'/>. FYI. (Stage: Running upgrade / reinstall script on oVirt Node)
2013-01-23 13:45:07,895 ERROR [org.ovirt.engine.core.bll.InstallerMessages] (NioProcessor-2) Installation of Host. Received illegal XML from Host. Message: org.ovirt.engine.core.compat.CompatException: Element type "BSTRAP" must be followed by either attribute specifications, ">" or "/>"., Exception: {2}
2013-01-23 13:45:07,896 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-2) Installation of 10.140.66.30. Received message: <BSTRAP component='setMountPoint' status='OK' message='Mount succeeded.'/>. FYI. (Stage: Running upgrade / reinstall script on oVirt Node)
2013-01-23 13:45:07,909 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-2) Installation of 10.140.66.30. Received message: <BSTRAP component='RHEV_INSTALL' status='FAIL'/>. Error occured. (Stage: Running upgrade / reinstall script on oVirt Node)
2013-01-23 13:45:07,915 ERROR [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-3-thread-49) SSH error running command 10.140.66.30:'/usr/share/vdsm-reg/vdsm-upgrade': java.io.IOException: Command returned failure code 1 during SSH session 10.140.66.30:22' '/usr/share/vdsm-reg/vdsm-upgrade'

The /var/log/vdsm-reg/vds_bootstrap_upgrade<date_time>.log file on the RHEV-H system showed;

Wed, 23 Jan 2013 13:45:07 DEBUG    <BSTRAP component='RHEL_INSTALL' status='OK'message='vdsm daemon stopped for upgrade process!'/>
Wed, 23 Jan 2013 13:45:07 DEBUG    ['/bin/umount', '/live']
Wed, 23 Jan 2013 13:45:07 DEBUG    
Wed, 23 Jan 2013 13:45:07 DEBUG    
Wed, 23 Jan 2013 13:45:07 DEBUG    <BSTRAP component='umount' status='OK' message='umount Succeeded'/>
Wed, 23 Jan 2013 13:45:07 DEBUG    ['/bin/mount', '-o', 'loop', '/data/updates/ovirt-node-image.iso', '/live']
Wed, 23 Jan 2013 13:45:07 DEBUG    
Wed, 23 Jan 2013 13:45:07 DEBUG    
Wed, 23 Jan 2013 13:45:07 DEBUG    <BSTRAP component='setMountPoint' status='OK' message='Mount succeeded.'/>
Wed, 23 Jan 2013 13:45:07 INFO     Using default value for: BOOT_SIZE
Wed, 23 Jan 2013 13:45:07 INFO     Using default value for: ROOT_SIZE
Wed, 23 Jan 2013 13:45:07 INFO     Using default value for: CONFIG_SIZE
Wed, 23 Jan 2013 13:45:07 INFO     Using default value for: LOGGING_SIZE
Wed, 23 Jan 2013 13:45:07 INFO     Using default value for: DATA_SIZE
Wed, 23 Jan 2013 13:45:07 INFO     Using default value for: SWAP2_SIZE
Wed, 23 Jan 2013 13:45:07 INFO     Using default value for: DATA2_SIZE
Wed, 23 Jan 2013 13:45:07 ERROR    <BSTRAP component='RHEV_INSTALL' status='FAIL'/>

The state of the system was that the host was in an "Installed Failed" state in the RHEV-M GUI, VDSM was not running and the RHEV-H ISO image was still mounted on the host.