Why does rhevm-upgrade from 3.1 to 3.2 fail with message "Error: Can't create trust store"?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Virtualization 3.1 to 3.2 upgrade

Issue

  • Trying to upgrade from RHEV 3.1 to RHEV 3.2 fails with the following message:
Would you like to proceed? (yes|no): yes
Stopping ovirt-engine service...                     [ DONE ]
Stopping DB related services...                      [ DONE ]
Cleaning async tasks...                              [ DONE ]
Pre-upgrade validations...                           [ DONE ]
Backing Up Database...                               [ DONE ]
Rename Database...                                   [ DONE ]
Updating rpms...                                     [ DONE ]
Updating Database...                                 [ DONE ]
Restore Database name...                             [ DONE ]
Preparing CA...                                      [ ERROR ]

 **Error: Upgrade failed, rolling back**
 **Reason: Error: Can't create trust store**

Restoring CA...                                      [ DONE ]
Restoring Database...                                [ DONE ]
Rolling back rpms...
Starting ovirt-engine service...                     [ DONE ]
Error: Upgrade failed.
please check log at /var/log/ovirt-engine/ovirt-engine-upgrade_2013_07_08_10_05_10.log

Resolution

  • Removed IBM version of Java.

  • If only IBM version is seen, remove it and install OpenJDK version in it's place.

Root Cause

  • IBM version of Java was preventing keyytool from performing the keystore generation.

Diagnostic Steps

  • Can see the failure in the upgrade log.
2013-07-08 10:07:32::DEBUG::common_utils::492::root:: stderr = 
2013-07-08 10:07:32::DEBUG::common_utils::493::root:: retcode = 0
2013-07-08 10:07:32::DEBUG::common_utils::1344::root:: Service was not stopped. there for we're not starting it
2013-07-08 10:07:32::DEBUG::rhevm-upgrade::473::root:: PKI: convert JKS to PKCS#12
2013-07-08 10:07:32::DEBUG::common_utils::453::root:: Executing command --> '/usr/bin/keytool -importkeystore -noprompt -srckeystore /etc/pki/ovirt-engine/.keystore -srcstoretype JKS -srcstorepass mypass -srcalias engine -srckeypass mypass -destkeystore /etc/pki/ovirt-engine/keys/engine.p12 -deststoretype PKCS12 -deststorepass mypass -destalias 1 -destkeypass mypass' in working directory '/'
2013-07-08 10:07:36::DEBUG::common_utils::491::root:: output = 
2013-07-08 10:07:36::DEBUG::common_utils::492::root:: stderr = 
2013-07-08 10:07:36::DEBUG::common_utils::493::root:: retcode = 0
2013-07-08 10:07:36::DEBUG::common_utils::1384::root:: chown /etc/pki/ovirt-engine/keys/engine.p12 to 108:108
2013-07-08 10:07:36::DEBUG::rhevm-upgrade::498::root:: PKI: dup cert for apache
2013-07-08 10:07:36::DEBUG::common_utils::748::root:: successfully copied file /etc/pki/ovirt-engine/keys/engine.p12 to target destination /etc/pki/ovirt-engine/keys/apache.p12
2013-07-08 10:07:36::DEBUG::common_utils::756::root:: setting file /etc/pki/ovirt-engine/keys/apache.p12 uid/gid ownership
2013-07-08 10:07:36::DEBUG::common_utils::759::root:: setting file /etc/pki/ovirt-engine/keys/apache.p12 mode to 416
2013-07-08 10:07:36::DEBUG::rhevm-upgrade::498::root:: PKI: dup cert for apache
2013-07-08 10:07:36::DEBUG::common_utils::748::root:: successfully copied file /etc/pki/ovirt-engine/certs/engine.cer to target destination /etc/pki/ovirt-engine/certs/apache.cer
2013-07-08 10:07:36::DEBUG::common_utils::756::root:: setting file /etc/pki/ovirt-engine/certs/apache.cer uid/gid ownership
2013-07-08 10:07:36::DEBUG::common_utils::759::root:: setting file /etc/pki/ovirt-engine/certs/apache.cer mode to 416
2013-07-08 10:07:36::DEBUG::rhevm-upgrade::498::root:: PKI: dup cert for apache
2013-07-08 10:07:36::DEBUG::common_utils::748::root:: successfully copied file /etc/pki/ovirt-engine/keys/engine_id_rsa to target destination /etc/pki/ovirt-engine/keys/apache.key.nopass
2013-07-08 10:07:36::DEBUG::common_utils::756::root:: setting file /etc/pki/ovirt-engine/keys/apache.key.nopass uid/gid ownership
2013-07-08 10:07:36::DEBUG::common_utils::759::root:: setting file /etc/pki/ovirt-engine/keys/apache.key.nopass mode to 416
2013-07-08 10:07:36::DEBUG::rhevm-upgrade::512::root:: PKI: dup ca for apache
2013-07-08 10:07:36::DEBUG::rhevm-upgrade::524::root:: PKI: dup cert for jboss
2013-07-08 10:07:36::DEBUG::common_utils::748::root:: successfully copied file /etc/pki/ovirt-engine/keys/apache.p12 to target destination /etc/pki/ovirt-engine/keys/jboss.p12
2013-07-08 10:07:36::DEBUG::common_utils::756::root:: setting file /etc/pki/ovirt-engine/keys/jboss.p12 uid/gid ownership
2013-07-08 10:07:36::DEBUG::common_utils::759::root:: setting file /etc/pki/ovirt-engine/keys/jboss.p12 mode to 416
2013-07-08 10:07:36::DEBUG::common_utils::1384::root:: chown /etc/pki/ovirt-engine/keys/jboss.p12 to 108:108
2013-07-08 10:07:36::DEBUG::rhevm-upgrade::542::root:: Converting truststore
2013-07-08 10:07:36::DEBUG::common_utils::453::root:: Executing command --> '/usr/bin/keytool -import -noprompt -keystore /etc/pki/ovirt-engine/.truststore.tmp -storepass ******** -keypass ******** -alias cacert -trustcacerts -file /etc/pki/ovirt-engine/ca.pem' in working directory '/'
2013-07-08 10:07:37::DEBUG::common_utils::491::root:: output = keytool error: java.lang.Exception: Input not an X.509 certificate

2013-07-08 10:07:37::DEBUG::common_utils::492::root:: stderr = 
2013-07-08 10:07:37::DEBUG::common_utils::493::root:: retcode = 1
2013-07-08 10:07:37::ERROR::rhevm-upgrade::1337::root:: Traceback (most recent call last):
  File "/usr/bin/rhevm-upgrade", line 1331, in main
    runFunc([ca.prepare], MSG_INFO_PKI_PREPARE)
  File "/usr/bin/rhevm-upgrade", line 649, in runFunc
    func()
  File "/usr/bin/rhevm-upgrade", line 556, in prepare
    utils.execCmd(cmdList=cmd, maskList=mask, failOnError=True, msg=MSG_ERROR_FAILED_CREATE_TRUSTSTORE)
  File "/usr/share/ovirt-engine/scripts/common_utils.py", line 496, in execCmd
    raise Exception(msg)
Exception: Error: Can't create trust store
  • Failure is at this point:
Executing command --> /usr/bin/keytool -import -noprompt -keystore /etc/pki/ovirt-engine/.truststore.tmp -storepass ******** -keypass ******** -alias cacert -trustcacerts -file /etc/pki/ovirt-engine/ca.pem and then this happens output = keytool error: java.lang.Exception: Input not an X.509 certificate.
  • Verified what version of Java is installed. Seen here are the two brands,IBM and OpenJDK, installed.
# rpm -qa |grep java
tzdata-java-2013c-2.el6.noarch
java_cup-0.10k-5.el6.x86_64
ovirt-host-deploy-java-1.0.0-2.el6ev.noarch
java-1.7.0-openjdk-1.7.0.25-2.3.10.4.el6_4.x86_64
glassfish-javamail-1.4.5-1.redhat_1.ep6.el6.noarch
java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64
otopi-java-1.0.0-2.el6ev.noarch
javassist-eap6-3.15.0-5.GA_redhat_2.ep6.el6.3.noarch
java-1.7.0-ibm-1.7.0.4.2-1jpp.1.el6_4.x86_64

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments