Issue

• Running a hosted-engine environment
• Power outage or some other unexpected event uncleanly shuts down RHEV Manager VM or HE host
• hosted-engine --vm-start fails to bring the RHEV Manager back online
• /var/log/vdsm/vdsm.log reports erros like this:

Thread-126::DEBUG::2014-07-04 11:17:22,613::libvirtconnection::108::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 38 edom: 42 level: 2 message: Failed to acquire lock: No space left on device
Thread-126::DEBUG::2014-07-04 11:17:22,614::vm::2178::vm.Vm::(_startUnderlyingVm) vmId=`af8e3fae-d19f-4e3e-919c-9f047440f072`::_ongoingCreations released
Thread-126::ERROR::2014-07-04 11:17:22,614::vm::2204::vm.Vm::(_startUnderlyingVm) vmId=`af8e3fae-d19f-4e3e-919c-9f047440f072`::The vm start process failed
Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 2164, in _startUnderlyingVm
    self._run()
  File "/usr/share/vdsm/vm.py", line 3121, in _run
    self._connection.createXML(domxml, flags),
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2665, in createXML
    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirtError: Failed to acquire lock: No space left on device
Thread-126::DEBUG::2014-07-04 11:17:22,693::vm::2658::vm.Vm::(setDownStatus) vmId=`af8e3fae-d19f-4e3e-919c-9f047440f072`::Changed state to Down: Failed to acquire lock: No space left on device

• /var/log/ovirt-hosted-engine-ha/agent.log reports errors like this:

MainThread::INFO::2014-07-04 15:00:45,718::hosted_engine::454::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /rhev/data-center/mnt/readynas.example.com:_isis/4b566b6f-9051-4993-88b0-a2315d7d2c40/ha_agent/hosted-engine.lockspace)
MainThread::ERROR::2014-07-04 15:00:46,719::hosted_engine::482::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) cannot get lock on host id 1: (-223, 'Sanlock lockspace add failure', 'Sanlock exception')
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 458, in _initialize_sanlock
    host_id, lease_file)
SanlockException: (-223, 'Sanlock lockspace add failure', 'Sanlock exception')
MainThread::WARNING::2014-07-04 15:00:46,720::hosted_engine::333::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Error while monitoring engine: Failed to initialize sanlock: cannot get lock on host id 1: (-223, 'Sanlock lockspace add failure', 'Sanlock exception')
MainThread::WARNING::2014-07-04 15:00:46,720::hosted_engine::336::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Unexpected error
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 322, in start_monitoring
    self._initialize_sanlock()
  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 484, in _initialize_sanlock
    .format(msg))
Exception: Failed to initialize sanlock: cannot get lock on host id 1: (-223, 'Sanlock lockspace add failure', 'Sanlock exception')
MainThread::INFO::2014-07-04 15:00:46,721::hosted_engine::353::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Sleeping 60 seconds