[RHEV]Failed to migrate VM between hypervisor with error "gaierror: [Errno -2] Name or service not known" in vdsm.log

Issue

• All VM are running on hypervisor rhevh02.xxxx.com, but they are failed to migrate to another hypervisor rhev01.xxxx.com.

• Events tab in RHEV_M shows below error.

2012-Nov-19, 12:41:42 Migration failed due to Error: Could not connect to peer host (VM: name, Source Host: rhevh02). Trying to migrate to another Host.

2012-Nov-19, 12:41:42 Migration failed due to Error: Could not connect to peer host (VM: name, Source Host: rhevh02).
rhevm.log while migration

• rhevm.log shows below error message.

2012-11-19 12:41:42,596 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-19-thread-47) START, MigrateBrokerVDSCommand(vdsId = 12a42a54-2f35-11e2-9833-00145e36327e, vmId=49f12334-4a22-45b3-b458-be122c483c27, srcHost=rhel-xxxx.xxx, dstVdsId=952496ea-2f34-11e2-b9d5-00145e36327e, dstHost=rhevh01.xxxx.com:54321, migrationMethod=ONLINE), log id: cbc07e2
2012-11-19 12:41:42,601 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-19-thread-47) FINISH, MigrateBrokerVDSCommand, log id: cbc07e2
2012-11-19 12:41:42,614 INFO  [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-19-thread-47) FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 4ef56479
2012-11-19 12:41:42,668 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-35) Rerun vm 49f12334-4a22-45b3-b458-be122c483c27. Called from vds rhevh02
2012-11-19 12:41:42,676 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-19-thread-47) START, MigrateStatusVDSCommand(vdsId = 12a42a54-2f35-11e2-9833-00145e36327e, vmId=49f12334-4a22-45b3-b458-be122c483c27), log id: 3de0557e
2012-11-19 12:41:42,679 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-19-thread-47) Failed in MigrateStatusVDS method
2012-11-19 12:41:42,680 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-19-thread-47) Error code noConPeer and error message VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Could not connect to peer VDS
2012-11-19 12:41:42,680 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-19-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value
 Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc
mStatus                       Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc
mCode                         10
mMessage                      Could not connect to peer VDS


2012-11-19 12:41:42,680 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-19-thread-47) Vds: Hyp2
2012-11-19 12:41:42,680 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-19-thread-47) Command MigrateStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Could not connect to peer VDS
2012-11-19 12:41:42,680 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-19-thread-47) FINISH, MigrateStatusVDSCommand, log id: 3de0557e
2012-11-19 12:41:42,687 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-19-thread-47) START, UpdateVdsDynamicDataVDSCommand(vdsId = 952496ea-2f34-11e2-b9d5-00145e36327e, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@4696c6d5), log id: 45471a83
2012-11-19 12:41:42,692 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-19-thread-47) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 45471a83
2012-11-19 12:41:42,711 WARN  [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-19-thread-47) CanDoAction of action MigrateVm failed. Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM

• /var/log/libvirtd.log on hypervisor rhev02

Thread-2990::ERROR::2012-09-05 03:00:55,541::vm::232::vm.Vm::(run) vmId=`4b761a43-09c2-4e5d-8da3-f99f0f38014e`::Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 224, in run
  File "/usr/share/vdsm/libvirtvm.py", line 423, in _startUnderlyingMigration
  File "/usr/share/vdsm/libvirtvm.py", line 445, in f
  File "/usr/share/vdsm/libvirtconnection.py", line 63, in wrapper
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1039, in migrateToURI
libvirtError: Unable to resolve address 'rhevh01' service '49152': Name or service not known

• vdsm.log on source hypervisor shows below error message.

Thread-724::ERROR::2012-11-19 12:39:31,861::vm::131::vm.Vm::(_setupVdsConnection) vmId=`49f12334-4a22-45b3-b458-be122c483c27`::Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 126, in _setupVdsConnection
    status = self.destServer.getVmStats(self._vm.id)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/share/vdsm/kaxmlrpclib.py", line 133, in request
    h.endheaders()
  File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
    self._send_output()
  File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.6/httplib.py", line 739, in send
    self.connect()
  File "/usr/share/vdsm/kaxmlrpclib.py", line 169, in connect
    self.sock.connect((self.host, self.port))
  File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 181, in connect
    self.socket.connect(addr)
  File "<string>", line 1, in connect
gaierror: [Errno -2] Name or service not known