Hypervisor fails to connect to iscsi storage with the error : "iscsiadm: initiator reported error (8 - connection timed out)"

Issue

• Hypervisor is stuck in "Unassigned" state in RHEVM portal. Every time we try to activate the host it goes to "Non operational" state.

• Following errors were noticed in vdsm.log of the troublesome Hypervisor:

Thread-3672::DEBUG::2013-09-29 04:14:30,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/iscsiadm -m node -T iqn.2013-05.com.host.san:iscsi-san1 -I default -p 192.16.1.1:3260 --op=delete' (cwd None)
Thread-3672::DEBUG::2013-09-29 04:14:30,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
Thread-3672::ERROR::2013-09-29 04:14:30,760::hsm::2298::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2295, in connectStorageServer
  File "/usr/share/vdsm/storage/storageServer.py", line 341, in connect
  File "/usr/share/vdsm/storage/iscsi.py", line 135, in addIscsiNode
  File "/usr/share/vdsm/storage/iscsiadm.py", line 292, in node_login
IscsiNodeError: (8, ['Logging in to [iface: default, target: iqn.2013-05.com.host.san:iscsi-san1, portal: 192.16.1.1,3260] (multiple)'], ['iscsiadm: Could not login to [iface: default, target: iqn.2013-05.com.host.san:iscsi-san1, portal: 192.168.1.1,3260].', 'iscsiadm: initiator reported error (8 - connection timed out)', 'iscsiadm: Could not log into all portals'])
Thread-3679::DEBUG::2013-09-29 04:14:30,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/iscsiadm -m node -T iqn.2013-05.com.host.san:iscsi-san1 -I default -p 192.16.1.1:3260 --op=new' (cwd None)
Thread-3672::DEBUG::2013-09-29 04:14:30,762::hsm::2310::Storage.HSM::(connectStorageServer) knownSDs: {}
Thread-3672::INFO::2013-09-29 04:14:30,797::logUtils::42::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'cd2115da-7a8d-4d30-be32-d3a5e66b4b17'}, {'status': 465, 'id': '1f27d497-a3db-4147-adb3-9ea49dc35a81'}]}
Thread-3672::DEBUG::2013-09-29 04:14:30,798::task::1168::TaskManager.Task::(prepare) Task=`f9e4483d-ff10-4aa6-8ffe-6fe28b5b5051`::finished: {'statuslist': [{'status': 0, 'id': 'cd2115da-7a8d-4d30-be32-d3a5e66b4b17'}, {'status': 465, 'id': '1f27d497-a3db-4147-adb3-9ea49dc35a81'}]}
Thread-3672::DEBUG::2013-09-29 04:14:30,799::task::579::TaskManager.Task::(_updateState) Task=`f9e4483d-ff10-4aa6-8ffe-6fe28b5b5051`::moving from state preparing -> state finished
Thread-3672::DEBUG::2013-09-29 04:14:30,799::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-3672::DEBUG::2013-09-29 04:14:30,799::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-3672::DEBUG::2013-09-29 04:14:30,799::task::974::TaskManager.Task::(_decref) Task=`f9e4483d-ff10-4aa6-8ffe-6fe28b5b5051`::ref 0 aborting False
Thread-3672::ERROR::2013-09-29 04:14:30,800::SecureXMLRPCServer::77::root::(handle_error) client ('172.20.245.39', 53152)
Traceback (most recent call last):
  File "/usr/lib64/python2.6/SocketServer.py", line 560, in process_request_thread
  File "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py", line 73, in finish_request
  File "/usr/lib64/python2.6/SocketServer.py", line 322, in finish_request
  File "/usr/lib64/python2.6/SocketServer.py", line 617, in __init__
  File "/usr/lib64/python2.6/BaseHTTPServer.py", line 329, in handle
  File "/usr/lib64/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
  File "/usr/lib64/python2.6/SimpleXMLRPCServer.py", line 494, in do_POST
  File "/usr/lib64/python2.6/socket.py", line 324, in write
  File "/usr/lib64/python2.6/socket.py", line 303, in flush
  File "/usr/lib64/python2.6/ssl.py", line 203, in sendall
  File "/usr/lib64/python2.6/ssl.py", line 174, in send

• In the RHEVM event logs for the troublesome hypervisor following errors were noticed :

networks are not synchronized with their logical network configuration