A RHEV host is non-responsive with VDSM reporting that a socket is "Address already in use".

Issue

• Host in Non-Responsive state.

• Engine logs contain;

2016-08-18 21:57:15,664 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command GetCapabilitiesVDSCommand(HostName = Host-A, HostId = 2e5acdc5-fec1-446c-bb7f-539cce14a6a0, vds=Host[Host-A,2e5acdc5-fec1-446c-bb7f-539cce14a6a0]) execution failed. Exception: VDSNetworkException: java.net.SocketException

• VDSM logs contain;

MainThread::ERROR::2016-08-18 15:46:42,272::vdsm::134::vds::(run) Exception raised
Traceback (most recent call last):
  File "/usr/share/vdsm/vdsm", line 132, in run
    serve_clients(log)
  File "/usr/share/vdsm/vdsm", line 82, in serve_clients
    cif = clientIF.getInstance(irs, log)
  File "/usr/share/vdsm/clientIF.py", line 158, in getInstance
  File "/usr/share/vdsm/clientIF.py", line 112, in __init__
  File "/usr/share/vdsm/clientIF.py", line 164, in _createAcceptor
  File "/usr/share/vdsm/protocoldetector.py", line 63, in __init__
  File "/usr/share/vdsm/protocoldetector.py", line 258, in _create_socket
  File "<string>", line 1, in bind
error: [Errno 98] Address already in use

• More than one instance of VDSM exists on the host;

vdsm     10369  0.0  0.0 6946980 139400 ?      S<   Aug05   0:00 /usr/bin/python /usr/share/vdsm/vdsm --pidfile /var/run/vdsm/vdsmd.pid

vdsm     26491  3.0  0.0 2670844 114400 ?      S<l  16:15   2:12 /usr/bin/python /usr/share/vdsm/vdsm --pidfile /var/run/vdsm/vdsmd.pid

• vdsClient commands failed;

# vdsClient -s 0 list table
   File "/usr/lib/python2.6/site-packages/vdsm/SecureXMLRPCServer.py", line 99, in connect
   File "/usr/lib64/python2.6/ssl.py", line 342, in wrap_socket
   File "/usr/lib64/python2.6/ssl.py", line 120, in __init__
   File "/usr/lib64/python2.6/ssl.py", line 279, in do_handshake
 error: [Errno 104] Connection reset by peer