Red Hat Enterprise Virtualization VM Migration fails immediately after starting with 'Connection refused' error

Issue

• Red Hat Enterprise Virtualization VM Migration fails immediately after starting migration with message Migration failed.

• On the source host, the vdsm logs contained a Connection refused error;

Thread-178624::ERROR::2016-09-13 19:22:17,950::migration::313::virt.vm::(run) vmId=`e2266c2d-e4dd-492d-bf53-4f5c2ee025db`::Failed to migrate
Traceback (most recent call last):
  File "/usr/share/vdsm/virt/migration.py", line 282, in run
    self._setupVdsConnection()
  File "/usr/share/vdsm/virt/migration.py", line 147, in _setupVdsConnection
    client = self._createClient(port)
  File "/usr/share/vdsm/virt/migration.py", line 134, in _createClient
    self.remoteHost, int(port), sslctx)
  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1269, in create_connected_socket
    sock.connect((host, port))
  File "/usr/lib64/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 181, in connect
    self.socket.connect(addr)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused

• vdsClient commands failed on the destination hosts;

# vdsClient -s 0 list table
Connection to 0.0.0.0:54321 refused

• All migrations to specific hosts would fail in this manner. Migrations from those hosts to the source host of the failing migrations were successful.

• Connections to port 54321 were failing on the destination hosts. However, RHEV-M was still connected to these hosts via that port, and those hosts were in an Up state.