[RHV] Host does not work as Hosted Engine host any more.

Solution Unverified - Updated -

Environment

  • Red Hat Virtualization 4.1

Issue

  • Red Hat Virtualization Host does not work as Hosted Engine host any more.

  • Error observed

RequestError: Request failed: failed to read metadata: [Errno 2] No such file or directory: '/rhev/data-center/mnt/blockSD/edxxxxxxxxx10c/ha_agent/hosted-engine.metadata'

Resolution

  • Place the cluster into global maintenance
# hosted-engine --set-maintenance --mode=global
  • Stop the HA agent and broker
# systemctl stop ovirt-ha-agent
# systemctl stop ovirt-ha-broker
  • Change ownership and permissions under /rhev path
 # chown -R 36:36 /rhev/data-center/mnt/blockSD/edxxxxxxxx010c/ha_agent
 # chmod -R 0770 /rhev/data-center/mnt/blockSD/edxxxxxxxxxx10c/ha_agent
  • Start the HA agent and broker
# systemctl start ovirt-ha-broker
# systemctl start ovirt-ha-agent
  • Monitor the environment and then remove the global maintenance 
# hosted-engine --set-maintenance --mode=none

Root Cause

  • ha_agent directory under /rhev/data-center/mnt/blockSD/edxxxxxxxxx10c/ shows ownership as root.
  • It should be owned by vdsm:kvm user:group and permissions should be 770.
  • Root cause of change in directory permission and ownership is unknown and not reproducible.

Diagnostic Steps

  • Observe "Permission denied" message while trying to create the symlink 'hosted-engine.lockspace'
Thread-3::ERROR::2018-06-01 12:34:17,679::listener::192::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Error handling request, data: 'set-storage-domain VdsmBackend hosted-engine.lockspace=7B226xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx661353665227D sp_uuid=0xxxx-xxxx-xxxx00 dom_type=fc hosted-engine.metadata=7B22696Dxxxxxxxxxxxx0343538632D303439xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx163333338343037227D sd_uuid=edxxxxxxxxxxx10c'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 166, in handle
    data)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 299, in _dispatch
    .set_storage_domain(client, sd_type, **options)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 66, in set_storage_domain
    self._backends[client].connect()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 415, in connect
    self._check_symlinks(self._storage_path, volume.path, service_link)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 133, in _check_symlinks
    str(e)
Exception: '/var/run/vdsm/storage/exxxxxxxxxx0c/45xxxxxx6d/dxxxxxxxx6e' -> '/rhev/data-center/mnt/blockSD/exxxxxxxxxx0c/ha_agent/hosted-engine.lockspace' failed: '[Errno 13] Permission denied'

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.