Problem after upgrading RHEV 3.1 All In One install

Latest response

Hi guys,

(started this question at: https://access.redhat.com/discussion/all-one-aio-rhev-plans)

I run an All In One install of RHEV 3.1 (used the all-in-one plugin).  I did a 'yum upgrade' and a 'rhevm-upgrade', now the web interface is gone. I'm looking into this, but maybe I should not have ran the upgrade on an All In One install?

I did not turn to GSS because of: "Red Hat's Global Support Services will not accept issues reported on this tool" https://access.redhat.com/knowledge/solutions/265803

 

After a 'service ovirt-engine start' is done, below can be found in /var/log/ovirt-engine/server.log:

...

2013-01-16 15:11:25,963 ERROR [org.jboss.as.server.deployment] (MSC service thread 1-5) JBAS015892: Deployment unit processor org.jboss.as.web.deployment.ELExpressionFactoryProcessor@3504c501 unexpectedly threw an exception during undeploy phase POST_MODULE of deployment "engine.ear": java.lang.NoClassDefFoundError: org/jboss/el/cache/FactoryFinderCache
        at org.jboss.as.web.deployment.ELExpressionFactoryProcessor.undeploy(ELExpressionFactoryProcessor.java:113)
        at org.jboss.as.server.deployment.DeploymentUnitPhaseService.safeUndeploy(DeploymentUnitPhaseService.java:172) [jboss-as-server.jar:7.1.3.Final-redhat-4]
        at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:121) [jboss-as-server.jar:7.1.3.Final-redhat-4]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc.jar:1.0.2.GA-redhat-2]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc.jar:1.0.2.GA-redhat-2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea]
        at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]

2013-01-16 15:11:25,973 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-5) MSC000001: Failed to start service jboss.deployment.unit."engine.ear".POST_MODULE: org.jboss.msc.service.StartException in service jboss.deployment.unit."engine.ear".POST_MODULE: JBAS018733: Failed to process phase POST_MODULE of deployment "engine.ear"
        at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:123) [jboss-as-server.jar:7.1.3.Final-redhat-4]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc.jar:1.0.2.GA-redhat-2]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc.jar:1.0.2.GA-redhat-2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea]
        at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]
Caused by: java.lang.NoClassDefFoundError: org/jboss/el/cache/FactoryFinderCache
        at org.jboss.as.web.deployment.ELExpressionFactoryProcessor.deploy(ELExpressionFactoryProcessor.java:106)
        at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:116) [jboss-as-server.jar:7.1.3.Final-redhat-4]
        ... 5 more
Caused by: java.lang.ClassNotFoundException: org.jboss.el.cache.FactoryFinderCache from [Module "org.jboss.as.web:main" from local module loader @25b94dbc (roots: /usr/share/ovirt-engine/modules,/usr/share/jbossas/modules)]
        at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:190)
        at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:468)
        at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:456)
        at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
        at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:120)
        ... 7 more

2013-01-16 15:11:26,184 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 17) JBAS018559: Deployed "engine.ear"
2013-01-16 15:11:26,304 INFO  [org.jboss.as.controller] (Controller Boot Thread) JBAS014774: Service status report
JBAS014777:   Services which failed to start:      service jboss.deployment.unit."engine.ear".POST_MODULE: org.jboss.msc.service.StartException in service jboss.deployment.unit."engine.ear".POST_MODULE: JBAS018733: Failed to process phase POST_MODULE of deployment "engine.ear"

 

(update: s/engine.log/server.log/)

VP
Log in to join the conversation

Responses

Richard Beldin's picture Expert 900 points
16 January 2013 8:44 PM

I'm having this same problem after a reboot this morning but I don't have the all-in-one plugin on RHEV 3.1.
engine.log is not being updated.  

server.log now shows:

2013-01-16 13:30:04,769 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000001: Failed to start service jboss.deployment.unit."engine.ear".POST_MODULE: org.jboss.msc.service.StartException in service jboss.deployment.unit."engine.ear".POST_MODULE: JBAS018733: Failed to process phase POST_MODULE of deployment "engine.ear"
        at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:123) [jboss-as-server.jar:7.1.3.Final-redhat-4]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc.jar:1.0.2.GA-redhat-2]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc.jar:1.0.2.GA-redhat-2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea]
        at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]
Caused by: java.lang.NoClassDefFoundError: org/jboss/el/cache/FactoryFinderCache
        at org.jboss.as.web.deployment.ELExpressionFactoryProcessor.deploy(ELExpressionFactoryProcessor.java:106)
        at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:116) [jboss-as-server.jar:7.1.3.Final-redhat-4]
        ... 5 more
Caused by: java.lang.ClassNotFoundException: org.jboss.el.cache.FactoryFinderCache from [Module "org.jboss.as.web:main" from local module loader @5cbf5bb7 (roots: /usr/share/ovirt-engine/modules,/usr/share/jbossas/modules)]
        at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:190)
        at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:468)
        at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:456)
        at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
        at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:120)
        ... 7 more

2013-01-16 13:30:04,784 WARN  [org.jboss.as.dependency.private] (MSC service thread 1-2) JBAS018567: Deployment "deployment.engine.ear.restapi.war" is using a private module ("org.codehaus.jackson.jackson-mapper-asl:main") which may be changed or removed in future versions without notice.
2013-01-16 13:30:04,784 WARN  [org.jboss.as.dependency.private] (MSC service thread 1-2) JBAS018567: Deployment "deployment.engine.ear.restapi.war" is using a private module ("org.codehaus.jackson.jackson-core-asl:main") which may be changed or removed in future versions without notice.
2013-01-16 13:30:04,784 WARN  [org.jboss.as.dependency.private] (MSC service thread 1-2) JBAS018567: Deployment "deployment.engine.ear.restapi.war" is using a private module ("org.codehaus.jackson.jackson-mapper-asl:main") which may be changed or removed in future versions without notice.
2013-01-16 13:30:04,990 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 17) JBAS018559: Deployed "engine.ear"
2013-01-16 13:30:04,991 INFO  [org.jboss.as.controller] (Controller Boot Thread) JBAS014774: Service status report
JBAS014777:   Services which failed to start:      service jboss.deployment.unit."engine.ear".POST_MODULE: org.jboss.msc.service.StartException in service jboss.deployment.unit."engine.ear".POST_MODULE: JBAS018733: Failed to process phase POST_MODULE of deployment "engine.ear"
 

VP Expert 823 points
17 January 2013 7:37 AM

Then I guess you should create a case at GSS, but is this a coincidence?

DY Guru 5340 points
17 January 2013 12:32 PM

Just upgraded an internal setup from 3.1.0 to 3.1.1 and everything works.

Was this setup initially 3.0 or even 2.x, before it was upgraded to 3.1?

VP Expert 823 points
17 January 2013 7:33 PM

My setup is a RHEV 3.1 All-In-One installation that was done on 12th or 13th of december last year.

If I understand the error message correctly, engine.ear undeploy is giving some troubles. Is there a way to do this undeploy manually?

This is my understanding of the rhevm-upgrade: the first time the ovirt-engine is started, after 'rhevm-upgrade' was run, new versions of the applications are deployed in JBoss. Is this correct?

VP Expert 823 points
21 January 2013 8:14 AM

@Dan

Is this an option:

- create a backup of the RHEV-M database

- run 'rhevm-cleanup'

- run 'rhevm-setup'

- restore RHEV-M database

 

Considering this is an All-In-One installation: should I backup other data (like certificates)?

DY Guru 5340 points
21 January 2013 10:20 AM

I think it should work, but I wouldn't vouch for that without trying first

DY Guru 5340 points
21 January 2013 10:52 AM

So far nobody managed to reproduce this issue, which probably means you have something customized that we are not aware of. Were there any issues in the upgrade logs?

VP Expert 823 points
21 January 2013 3:31 PM

@Dan: I already have had some help from you on irc, but I want to share the latest state:

 

I did:

- yum remove rhevm rhev* jboss*

- reboot

- yum clean all

- yum install rhevm rhevm-setup-plugin-allinone

 

This should be a clean install, but when ovirt-engine is run, i still get the same failure in server.log (see my first post).

I wil try a clean full system install when I got some time, but something is 'buggy' ....

VP Expert 823 points
22 January 2013 10:03 AM

I managed to get things running again:

~$ /etc/init.d/ovirt-engine stop

~$ rhevm-clean

~$ yum remove rhevm rhev* jboss* qemu-kvm-rhev-tools qemu-img-rhev qemu-kvm-rhev slf4j-jboss-logmanager jacorb-jboss log4j-jboss-logmanager jaxws-jboss-httpserver-httpspi

(yes, run the remove commands multiple times)

~$ yum remove rhevm rhev* jboss* qemu-kvm-rhev-tools qemu-img-rhev qemu-kvm-rhev slf4j-jboss-logmanager jacorb-jboss log4j-jboss-logmanager jaxws-jboss-httpserver-httpspi

~$ yum remove rhevm rhev* jboss* qemu-kvm-rhev-tools qemu-img-rhev qemu-kvm-rhev slf4j-jboss-logmanager jacorb-jboss log4j-jboss-logmanager jaxws-jboss-httpserver-httpspi

~$ rm -Rf /etc/pki/ovirt-engine-backups /etc/ovirt-engine /usr/share/ovirt-engine /var/lib/ovirt-engine /etc/jbossas /usr/share/jbossas /var/lib/jbossas

~$ yum install rhevm rhevm-setup-plugin-allinone

~$ rhevm-setup (used temporary name for the local storage directory)

RHEV-M is now available again! Next stop ovirt-engine again and restore of the 'engine' DB

After starting ovirt-engine again, I tried to login with the 'admin' user, but nogo. The password (and other things) for the 'admin' user couldn't be decrypted anymore:

~# rhevm-config -g AdminPassword
Failed to decrypt Data must start with zero
Failed to decrypt
javax.crypto.BadPaddingException: Data must start with zero
    at sun.security.rsa.RSAPadding.unpadV15(RSAPadding.java:325)
    at sun.security.rsa.RSAPadding.unpad(RSAPadding.java:272)
    at com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:356)
    at com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:382)
    at javax.crypto.Cipher.doFinal(Cipher.java:1922)
    at org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:157)
    at org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:141)
    at org.ovirt.engine.core.config.entity.helper.PasswordValueHelper.decrypt(PasswordValueHelper.java:49)
    at org.ovirt.engine.core.config.entity.helper.PasswordValueHelper.getValue(PasswordValueHelper.java:65)
    at org.ovirt.engine.core.config.entity.ConfigKey.getDisplayValue(ConfigKey.java:73)
    at org.ovirt.engine.core.config.EngineConfigLogic.printAllValuesForKey(EngineConfigLogic.java:219)
    at org.ovirt.engine.core.config.EngineConfigLogic.printKey(EngineConfigLogic.java:291)
    at org.ovirt.engine.core.config.EngineConfigLogic.execute(EngineConfigLogic.java:104)
    at org.ovirt.engine.core.config.EngineConfig.setUpAndExecute(EngineConfig.java:37)
    at org.ovirt.engine.core.config.EngineConfig.main(EngineConfig.java:50)
Failed to decrypt the current value

The engine.log shows:

2013-01-22 10:47:44,533 WARN  [org.ovirt.engine.core.utils.ConfigUtilsBase] (ServerService Thread Pool -- 33) Could not find enum value for option: CbcCheckOnVdsChange
2013-01-22 10:47:44,541 WARN  [org.ovirt.engine.core.utils.ConfigUtilsBase] (ServerService Thread Pool -- 33) Could not find enum value for option: ENGINEEARLib
2013-01-22 10:47:44,543 WARN  [org.ovirt.engine.core.utils.ConfigUtilsBase] (ServerService Thread Pool -- 33) Could not find enum value for option: CAEngineKey
2013-01-22 10:47:44,554 WARN  [org.ovirt.engine.core.utils.ConfigUtilsBase] (ServerService Thread Pool -- 33) Could not find enum value for option: MinimalETLVersion
2013-01-22 10:47:44,605 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ServerService Thread Pool -- 33) Failed to decrypt Data must start with zero
2013-01-22 10:47:44,605 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ServerService Thread Pool -- 33) Failed to decrypt value for property LocalAdminPassword will be used encrypted value
2013-01-22 10:47:44,616 WARN  [org.ovirt.engine.core.utils.ConfigUtilsBase] (ServerService Thread Pool -- 33) Could not find enum value for option: ScriptsPath
2013-01-22 10:47:44,617 WARN  [org.ovirt.engine.core.utils.ConfigUtilsBase] (ServerService Thread Pool -- 33) Could not find enum value for option: SQLServerI18NPrefix
2013-01-22 10:47:44,840 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ServerService Thread Pool -- 33) Failed to decrypt Data must start with zero
2013-01-22 10:47:44,841 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ServerService Thread Pool -- 33) Failed to decrypt value for property TruststorePass will be used encrypted value
2013-01-22 10:47:44,872 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ServerService Thread Pool -- 33) Failed to decrypt Data must start with zero
2013-01-22 10:47:44,873 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ServerService Thread Pool -- 33) Failed to decrypt value for property CertificatePassword will be used encrypted value
2013-01-22 10:47:44,897 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ServerService Thread Pool -- 33) Failed to decrypt Data must start with zero
2013-01-22 10:47:44,897 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ServerService Thread Pool -- 33) Failed to decrypt value for property AdminPassword will be used encrypted value

Unfortunatly I can not reset the password:

[root@rhev31 dbscripts]# rhevm-config -s AdminPassword=VerySecret
Cannot set value VerySecret to key AdminPassword.

How to fix this?

VP Expert 823 points
22 January 2013 12:17 PM

Nevermind, I managed to login again (copied some password values from the 'fresh' vdc_options table), but can't activate the localstorage.

I probably missed some key / cert files when creating the backup. By doing a reinstall again and again files have changed and I can't reproduce the exact (backup) config any more.

 

Anyway, it was a very interesting journey :)

DY Guru 5340 points
22 January 2013 1:28 PM

 

> Cannot set value VerySecret to key AdminPassword.
 
if you run rhevm-config --help, you will notice there are some specifics mentioned:
 

rhevm-config --help

engine-config: get/set/list configuration

USAGE:

        engine-config ACTION [--cver=version] [-p | --properties=/path/to/alternate/property/file] [-c | --config=/path/to/alternate/config/file]

Where:

        ACTION              action to perform, see details below

        version             relevant configuration version to use.

        -p, --properties=   (optional) use the given alternate properties file.

        -c, --config=       (optional) use the given alternate configuration file.

        Available actions:

        -l, --list

                list available configuration keys.

        -a, --all

                get all available configuration values.

        -g key, --get=key [--cver=version]

                get the value of the given key for the given version. If a version is not given, the values of all existing versions are returned.

        -s key=val --cver=version, --set key=val --cver=version

                set the value of the given key for the given version. The cver version is mandatory for this action.

        -h, --help

                display this help and exit.

### Notes: 

### 1. Passwords: password can be set in interacetive mode ie:

###        engine-config -s PasswordEntry=interactive

###    or via file with one of the following options:

###        engine-config -s PasswordEntry --admin-pass-file=/tmp/mypass

###        engine-config -s PasswordEntry=/tmp/mypass

### 2. In order for your change(s) to take effect,

###    restart the oVirt engine service (using: 'service ovirt-engine restart').

################################################################################

I marked the important part in bold. Inserting passwords openly is not a good idea, and should never be done, so even in rhevm-config, we force it to be hidden

VP Expert 823 points
22 January 2013 2:02 PM

Thnx Dan!

 

But I think that command would fail also, I didn't create a complete backup. I agree: it is better to use the interactive mode.