Host stays in non responsive and ConnectStorageServerVDSCommand is timing out in the engine side

Issue

• The host is in non responsive status and ConnectStorageServerVDSCommand is getting timeout on the engine side.

2018-01-03 12:32:01,919+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler5) [f374610] START, ConnectStorageServerVDSCommand(HostName = node1, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='0a4d0814-1d4e-4284-baee-4318da8cedde', storagePoolId='724a7b91-09.....iqn.2001-05.com.equallogic:0-8a0906-f8d242a0c-4ad0002183054509-test-host-vol1', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='eth0', netIfaceName='iscsi2'}]'}), log id: f1fa6f

2018-01-03 12:35:02,034+08 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [f374610] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM sn1mopx2m0006 command ConnectStorageServerVDS failed: Message timeout which can be caused by communication issues

• vdsm shows the command as succesfull however it took more than 3 minutes to complete.

jsonrpc.Executor/1::INFO::2018-01-03 04:09:27,229::logUtils::48::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=3, spUUID=u'724a7b91-0927-4b3b-813b-402e93372878', conList=[{u'netIfaceName': u'iscsi1', u'id': u'd2f0188b-3ea4-4097-8758-d5359bd7171c', u'connection': u'10.0.0.1', u'iqn': u'iqn.2001-05.com.test:0-8a0906-03d242a0c-d580000a33652b96-test-host-vol1', u'user': u'', u'tpgt': u'1', u'ifaceName': u'eth0', u'password':
.....
.....
p6p1', u'password': '********', u'port': u'3260'}, {u'netIfaceName': u'iscsi2', u'id': u'6ca99202-326d-44cc-8814-6c6e41c3ba15', u'connection': u'10.0.0.2', u'iqn': u'iqn.2001-05.com.equallogic:0-8a0906-f8d242a0c-4ad0002183054509-test-host-vol2', u'user': u'', u'tpgt': u'1', u'ifaceName': u'p6p1', u'password': '********', u'port': u'3260'}], options=None)
jsonrpc.Executor/1::INFO::2018-01-03 04:12:55,832::__init__::514::jsonrpc.JsonRpcServer::(_serveRequest) RPC call StoragePool.connectStorageServer succeeded in 208.61 seconds

• The environment is having around 10 iscsi target with two paths. So a total of 20 iscsi session has to be connected.