Migrating VMs to a host or starting VMs on a host fail with "Unexpected exception", "unexpected error", "Permission denied"

Issue

• VM migrations failed to a specific host, but were then successful to other hosts.

• The VM migrations failed with the following errors;

1) In the Event Log in the RHEV Admin Portal;

Migration failed due to Error: Unexpected exception. Trying to migrate to another Host (VM: vm-1, Source: host-1, Destination: host-2).

2) In the engine logs on the RHEV Manager;

2014-05-05 14:08:32,534 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-12) [61286251] HostName = host-2
2014-05-05 14:08:32,535 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-12) [61286251] Command DestroyVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to DestroyVDS, error = Unexpected exception
2014-05-05 14:08:32,535 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-12) [61286251] FINISH, DestroyVDSCommand, log id: 3c55f134
2014-05-05 14:08:32,554 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [61286251] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM vm-1 is down. Exit message: Domain not found: no domain with matching uuid '8c985609-fb25-4c03-ae2f-d072304c9021'

3) In the vdsm logs on the source host;

Thread-133795::ERROR::2014-05-05 13:20:53,198::vm::239::vm.Vm::(_recover) vmId=`8c985609-fb25-4c03-ae2f-d072304c9021`::migration destination error: Unexpected exception

4) In the vdsm logs on the destination host;

Thread-134086::DEBUG::2014-05-05 13:20:53,180::vm::2143::vm.Vm::(_startUnderlyingVm) vmId=`8c985609-fb25-4c03-ae2f-d072304c9021`::_ongoingCreations released
Thread-134086::DEBUG::2014-05-05 13:20:53,181::vm::3564::vm.Vm::(_waitForIncomingMigrationFinish) vmId=`8c985609-fb25-4c03-ae2f-d072304c9021`::Waiting 21600 seconds for end of migration
Thread-134085::ERROR::2014-05-05 13:20:53,194::BindingXMLRPC::994::vds::(wrapper) unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/BindingXMLRPC.py", line 980, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/BindingXMLRPC.py", line 335, in vmMigrationCreate
    return vm.migrationCreate(params)
  File "/usr/share/vdsm/API.py", line 514, in migrationCreate
    if not v.waitForMigrationDestinationPrepare():
  File "/usr/share/vdsm/vm.py", line 5084, in waitForMigrationDestinationPrepare
    hooks.before_vm_migrate_destination(srcDomXML, self.conf)
  File "/usr/share/vdsm/hooks.py", line 202, in before_vm_migrate_destination
    return _runHooksDir(domxml, 'before_vm_migrate_destination', vmconf=vmconf)
  File "/usr/share/vdsm/hooks.py", line 62, in _runHooksDir
    data_fd, data_filename = tempfile.mkstemp()
  File "/usr/lib64/python2.6/tempfile.py", line 293, in mkstemp
    return _mkstemp_inner(dir, prefix, suffix, flags)
  File "/usr/lib64/python2.6/tempfile.py", line 228, in _mkstemp_inner
    fd = _os.open(file, flags, 0600)
OSError: [Errno 13] Permission denied: '/tmp/tmpryTMER'

Followed by many of the these;

Thread-133684::ERROR::2014-05-05 13:09:11,761::BindingXMLRPC::994::vds::(wrapper) unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/BindingXMLRPC.py", line 980, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/BindingXMLRPC.py", line 201, in vmDestroy
    return vm.destroy()
  File "/usr/share/vdsm/API.py", line 329, in destroy
    res = v.destroy()
  File "/usr/share/vdsm/vm.py", line 4506, in destroy
    response = self.releaseVm()
  File "/usr/share/vdsm/vm.py", line 4476, in releaseVm
    hooks.after_vm_destroy(self._lastXMLDesc, self.conf)
  File "/usr/share/vdsm/hooks.py", line 236, in after_vm_destroy
    raiseError=False)
  File "/usr/share/vdsm/hooks.py", line 62, in _runHooksDir
    data_fd, data_filename = tempfile.mkstemp()
  File "/usr/lib64/python2.6/tempfile.py", line 293, in mkstemp
    return _mkstemp_inner(dir, prefix, suffix, flags)
  File "/usr/lib64/python2.6/tempfile.py", line 228, in _mkstemp_inner
    fd = _os.open(file, flags, 0600)
OSError: [Errno 13] Permission denied: '/tmp/tmppRJytC'

• Starting a VM can also fail with the same errors. In this case, the Event Log in the Admin Portal would show something similar to the following;

VM vm-1 is down. Exit message: [Errno 13] Permission denied: '/tmp/tmpL0Tw4E'.