Unable to add hypervisor host to clusters after upgrading from RHEV-M 3.4.1 to 3.4.5-0.3
Environment
- Red Hat Enterprise Virtualization (RHEV) 3.45-0.3 (upgraded from 3.4.1)
- Red Hat Enterprise Linux 6.6 Hypervisor host.
Issue
- Unable to add hypervisor host to clusters after upgrading from RHEV-M 3.4.1 to 3.4.5-0.3.
- The hypervisor hosts changes to Non-Operational state.
Resolution
-
The /etc/pki/ovirt-engine/ca.pem file must have the root:root owner with 644 permission set.
-
On the RHEV-M, correct the file permission for /etc/pki/ovirt-engine/ca.pem
# cd /etc/pki/ovirt-engine/
# chown root:root ca.pem
# chmod 644 ca.pem
# service ovirt-engine restart
Root Cause
- Incorrect file permission set for /etc/pki/ovirt-engine/ca.pem.
-rw-r-----. 1 ovirt ovirt 4877 Nov 20 2013 ca.pem
Diagnostic Steps
- Error reported in the engine.log file
2015-06-02 10:45:48,645 ERROR [org.ovirt.engine.core.bll.VdsDeploy] (org.ovirt.thread.pool-13-thread-33) [147b9abd] Error during host RHEVHOST.example.com install: java.io.IOException: Command returned failure code 1 during SSH session 'root@RHEVHOST.example.com'
2015-06-02 10:45:48,647 ERROR [org.ovirt.engine.core.bll.InstallVdsCommand] (org.ovirt.thread.pool-13-thread-33) [147b9abd] Host installation failed for host 246754d7-e7e0-466d-87b8-8c8514a88a79, RHEVHOST.example.com.: java.io.IOException: Command returned failure code 1 during SSH session 'root@RHEVHOST.example..com'
2015-06-02 10:45:48,680 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-13-thread-33) [147b9abd] START, SetVdsStatusVDSCommand(HostName = RHEVHOST.example.com, HostId = 246754d7-e7e0-466d-87b8-8c8514a88a79,, status=InstallFailed, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4c580cfe
2015-06-02 10:45:48,684 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-13-thread-33) [147b9abd] FINISH, SetVdsStatusVDSCommand, log id: 4c580cfe
- From the ovirt-host-deploy log file, the install process fails to retrieve vdsm certificates from RHEV-M and thus it aborts waiting for proper certificates
2015-06-02 10:45:02 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:215 DIALOG:SEND -----END CERTIFICATE REQUEST-----
2015-06-02 10:45:02 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:215 DIALOG:SEND --=12345678-1234-1234-1234-123456789012=--
2015-06-02 10:45:02 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:215 DIALOG:SEND ***Q:MULTI-STRING VDSM_CERTIFICATE_CHAIN --=12345678-1234-1234-1234-123456789012=-- --=12345678-1234-1234-1234-123456789012=--
2015-06-02 10:45:02 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:215 DIALOG:SEND ###
2015-06-02 10:45:02 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:215 DIALOG:SEND ###
2015-06-02 10:45:02 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:215 DIALOG:SEND ### Please input VDSM certificate chain that matches certificate request, top is issuer
2015-06-02 10:45:02 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:215 DIALOG:SEND ###
2015-06-02 10:45:02 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:215 DIALOG:SEND ### type '--=12345678-1234-1234-1234-123456789012=--' in own line to mark end, '--=12345678-1234-1234-1234-123456789012=--' aborts
2015-06-02 10:45:11 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
File "/tmp/ovirt-zANqASQg4B/pythonlib/otopi/context.py", line 142, in _executeMethod
method['method']()
File "/tmp/ovirt-zANqASQg4B/otopi-plugins/ovirt-host-deploy/vdsm/pki.py", line 316, in _misc
'\n\nPlease input VDSM certificate chain that '
File "/tmp/ovirt-zANqASQg4B/otopi-plugins/otopi/dialog/machine.py", line 204, in queryMultiString
v = self._readline()
File "/tmp/ovirt-zANqASQg4B/pythonlib/otopi/dialog.py", line 259, in _readline
raise IOError(_('End of file'))
IOError: End of file
- When attempting to get the ca.crt and engine.ssh.key.txt using the wget https://
/ca.crt --no-check-certificate and https:///engine.ssh.key.txt --no-check-certificate , the message ERROR 503: Service Unavailable. error is reported
# wget https://<RHEV-M-FQDN>/ca.crt --no-check-certificate
--2015-06-12 10:00:05-- https://<RHEV-M-FQDN>/ca.crt
Resolving RHEV-M-FQDN .. xx.xxx.xxx.xxx
Connecting to RHEV-M-FQDN|xx.xxx.xx.xxx|:xxx... connected.
WARNING: cannot verify RHEV-M-FQDN's certificate, issued by “/C=US/O=Some_Company./CN=CA-RHEV-M-FQDN.xxxxx”:
Self-signed certificate encountered.
HTTP request sent, awaiting response... 503 Service Unavailable
2015-06-12 10:00:05 ERROR 503: Service Unavailable.
# wget https://<RHEV-M-FQDN>/engine.ssh.key.txt --no-check-certificate
--2015-06-12 09:34:11-- https://<RHEV-M-FQDN>/engine.ssh.key.txt
Resolving RHEV-M-FQDN .. xx.xxx.xxx.xxx
Connecting to RHEV-M-FQDN|xx.xxx.xx.xxx|:xxx... connected.
WARNING: cannot verify RHEV-M-FQDN's certificate, issued by “/C=US/O=Some_Company./CN=CA-RHEV-M-FQDN.xxxxx”:
Self-signed certificate encountered.
HTTP request sent, awaiting response... 503 Service Unavailable
2015-06-12 09:34:11 ERROR 503: Service Unavailable.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
