The RHEV-M portal fails to load giving errors after the engine-setup command failed during RHEVM upgrade.

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Virtualization 3.5
  • Self- hosted RHEV

Issue

  • ovirt-engine throwing java errors after attempting to upgrade RHEVM versions from 3.5.0.0 to 3.5.7.0 with error as ERROR: function get_all_commands() does not exist
  • RHEV-M upgrade failed with below messages in the logs:
2016-01-15 16:09:28 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine_reports.jasper.deploy plugin.execute:937 execute-output: ('./js-export.sh', '--output-dir', '/tmp/tmpW1r2Ka/users', '--users', '--roles') stderr:

2016-01-15 16:09:28 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/otopi/context.py", line 142, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine-reports/jasper/deploy.py", line 638, in _export
    '--roles',
  File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/reports/reportsutil.py", line 129, in jsexport
    oreportscons.FileLocations.
  File "/usr/lib/python2.6/site-packages/otopi/plugin.py", line 942, in execute
    command=args[0],
RuntimeError: Command './js-export.sh' failed to execute
2016-01-15 16:09:28 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Misc configuration': Command './js-export.sh' failed to execute

Resolution

1] Put to RHEVM VM to global maintenance mode.

2] Then roll back the update with the below command:

# yum history undo <ID-of-Update-from-yum-history-cmd>

3] Stop the below services before proceeding with the rhev db restore:

# service ovirt-engine stop
# service ovirt-engine-dwhd stop
# service ovirt-engine-reportsd stop
# service ovirt-fence-kdump-listener stop

4] Empty the engine database :

 # su postgres -c psql
postgres=#  DROP DATABASE engine;
postgres=# CREATE DATABASE engine OWNER engine template template0 encoding 'UTF8' lc_collate 'en_US.UTF-8' lc_ctype 'en_US.UTF-8';
postgres-# \q

5] Restore db using below command:

# su - postgres 

   - Then  run the below command, to restore the db: 

# pg_restore -d engine /tmp/engine-20160115160530.gjv2JT.dump

    - This restored the db successfully. 

6] Further, restart the ovirt-engine service and check.

Root Cause

  • The upgrade did not complete because the js-export.sh failed to complete successfully, when something fails during update engine-setup will attempt to roll back, but engine-setup stops before it can successfully roll back.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.