Unable to migrate guests between hosts on Red Hat Enterprise Virtulization 3.0.

Issue

• While migrating guest, it fails with below error message.

  Migration failed due to Error: Could not connect to peer host (VM: Name, Source Host: Host-2).
  Migration failed due to Error: Could not connect to peer host (VM: Name, Source Host: Host-2).
  Migration failed due to Error: Could not connect to peer host (VM: Name, Source Host: Host-2).
  

• rhevm.log shows below messages:

2012-06-13 22:18:27,569 INFO  [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-20-thread-44) Running command: MigrateVmCommand internal: false. Entities affected :  ID: 90c7e045-b1e7-4f59-9385-846f4b1c3d95 Type: VM
2012-06-13 22:18:27,592 INFO  [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-20-thread-44) START, MigrateVDSCommand(vdsId = 3c22dd3a-b488-11e1-890d-525400459e91, vmId=90c7e045-b1e7-4f59-9385-846f4b1c3d95, srcHost=192.168.134.103, dstVdsId=c3f6bbc4-b590-11e1-ba21-525400459e91, dstHost=192.168.134.107:54321, migrationMethod=ONLINE), log id: 1aba563f
2012-06-13 22:18:27,597 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-20-thread-44) VdsBroker::migrate::Entered (vm_guid=90c7e045-b1e7-4f59-9385-846f4b1c3d95, srcHost=192.168.134.103, dstHost=192.168.134.107:54321,  method=online
2012-06-13 22:18:27,597 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-20-thread-44) START, MigrateBrokerVDSCommand(vdsId = 3c22dd3a-b488-11e1-890d-525400459e91, vmId=90c7e045-b1e7-4f59-9385-846f4b1c3d95, srcHost=192.168.134.103, dstVdsId=c3f6bbc4-b590-11e1-ba21-525400459e91, dstHost=192.168.134.107:54321, migrationMethod=ONLINE), log id: 15bc68c6
2012-06-13 22:18:27,609 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-20-thread-44) FINISH, MigrateBrokerVDSCommand, log id: 15bc68c6
2012-06-13 22:18:27,620 INFO  [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-20-thread-44) FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 1aba563f
2012-06-13 22:18:28,098 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-85) Rerun vm 90c7e045-b1e7-4f59-9385-846f4b1c3d95. Called from vds rhev3.example.net
2012-06-13 22:18:28,108 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-20-thread-46) START, MigrateStatusVDSCommand(vdsId = 3c22dd3a-b488-11e1-890d-525400459e91, vmId=90c7e045-b1e7-4f59-9385-846f4b1c3d95), log id: 15b91a78
2012-06-13 22:18:28,114 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-20-thread-46) Failed in MigrateStatusVDS method
2012-06-13 22:18:28,114 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-20-thread-46) Error code noConPeer and error message VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Could not connect to peer VDS
2012-06-13 22:18:28,115 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-20-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value
 Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc
mStatus                       Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc
mCode                         10
mMessage                      Could not connect to peer VDS

2012-06-13 22:18:28,115 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-20-thread-46) Vds: rhev3.example.net
2012-06-13 22:18:28,115 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-20-thread-46) Command MigrateStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Could not connect to peer VDS
2012-06-13 22:18:28,115 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-20-thread-46) FINISH, MigrateStatusVDSCommand, log id: 15b91a78
2012-06-13 22:18:28,118 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-20-thread-46) START, UpdateVdsDynamicDataVDSCommand(vdsId = c3f6bbc4-b590-11e1-ba21-525400459e91, vdsDynamic=org.ovirt.engine.core.common.businessentities.VdsDynamic@b1fa1295), log id: 29f7989e
2012-06-13 22:18:28,122 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVdsDynamicDataVDSCommand] (pool-20-thread-46) FINISH, UpdateVdsDynamicDataVDSCommand, log id: 29f7989e
2012-06-13 22:18:28,161 WARN  [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-20-thread-46) CanDoAction of action MigrateVm failed. Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM

• vdsm.log shows below messages:

Thread-89393::DEBUG::2012-06-13 20:18:27,601::clientIF::54::vds::(wrapper) [192.168.134.15]::call migrate with ({'src': '192.168.134.103', 'dst': '192.168.134.107:54321', 'vmId': '90c7e045-b1e7-4f59-9385-846f4b1c3d95', 'method': 'online'},) {}
Thread-89393::DEBUG::2012-06-13 20:18:27,602::clientIF::357::vds::(migrate) {'src': '192.168.134.103', 'dst': '192.168.134.107:54321', 'vmId': '90c7e045-b1e7-4f59-9385-846f4b1c3d95', 'method': 'online'}
Thread-89394::DEBUG::2012-06-13 20:18:27,605::vm::123::vm.Vm::(_setupVdsConnection) vmId=`90c7e045-b1e7-4f59-9385-846f4b1c3d95`::Destination server is: https://192.168.134.107:54321
Thread-89393::DEBUG::2012-06-13 20:18:27,605::clientIF::59::vds::(wrapper) return migrate with {'status': {'message': 'Migration process starting', 'code': 0}}
Thread-89394::DEBUG::2012-06-13 20:18:27,606::vm::125::vm.Vm::(_setupVdsConnection) vmId=`90c7e045-b1e7-4f59-9385-846f4b1c3d95`::Initiating connection with destination
Thread-89394::ERROR::2012-06-13 20:18:27,611::vm::131::vm.Vm::(_setupVdsConnection) vmId=`90c7e045-b1e7-4f59-9385-846f4b1c3d95`::Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 126, in _setupVdsConnection
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
  File "/usr/share/vdsm/kaxmlrpclib.py", line 133, in request
  File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
  File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
  File "/usr/lib64/python2.6/httplib.py", line 739, in send
  File "/usr/share/vdsm/kaxmlrpclib.py", line 169, in connect
  File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 185, in connect
  File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 178, in connect_ssl
SSLError: certificate verify failed

Thread-89394::DEBUG::2012-06-13 20:18:27,613::vm::171::vm.Vm::(_prepareGuest) vmId=`90c7e045-b1e7-4f59-9385-846f4b1c3d95`::migration Process begins
Thread-89394::DEBUG::2012-06-13 20:18:27,616::vm::218::vm.Vm::(run) vmId=`90c7e045-b1e7-4f59-9385-846f4b1c3d95`::migration semaphore acquired
Thread-89394::ERROR::2012-06-13 20:18:27,622::vm::177::vm.Vm::(_recover) vmId=`90c7e045-b1e7-4f59-9385-846f4b1c3d95`::certificate verify failed
Thread-89394::ERROR::2012-06-13 20:18:27,624::vm::182::vm.Vm::(_recover) vmId=`90c7e045-b1e7-4f59-9385-846f4b1c3d95`::Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 180, in _recover
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
  File "/usr/share/vdsm/kaxmlrpclib.py", line 133, in request
  File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
  File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
  File "/usr/lib64/python2.6/httplib.py", line 739, in send
  File "/usr/share/vdsm/kaxmlrpclib.py", line 169, in connect
  File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 185, in connect
  File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 178, in connect_ssl
SSLError: certificate verify failed

Thread-89394::ERROR::2012-06-13 20:18:27,627::vm::232::vm.Vm::(run) vmId=`90c7e045-b1e7-4f59-9385-846f4b1c3d95`::Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 224, in run
  File "/usr/share/vdsm/libvirtvm.py", line 397, in _startUnderlyingMigration
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
  File "/usr/share/vdsm/kaxmlrpclib.py", line 133, in request
  File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
  File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
  File "/usr/lib64/python2.6/httplib.py", line 739, in send
  File "/usr/share/vdsm/kaxmlrpclib.py", line 169, in connect
  File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 185, in connect
  File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 178, in connect_ssl
SSLError: certificate verify failed