Slow/Blocked EJB client when multiple JBoss EAP instances are configured as backend EJB servers in unstable network environment

Issue

When EJB client (either standalone Java EJB client, or another JBoss EAP server in server-to-server EJB calls) is configured to connect to multiple back-end EJB/EAP server instances, it is possible to see EJB client becomes slow or blocked, when one (or multiple) backend EJB node(s) drops off network, or has slow network connections.

Capturing thread dumps when EJB client becomes slow or blocked, we may see similar threads as below waiting on DiscoveryEJBClientInterceptor.doAnyDiscovery():

"http-executor task-3" #801 prio=5 os_prio=0 tid=0x0000556d51610800 nid=0xca8 waiting on condition [0x00007f8617b4b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000dc522618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.wildfly.discovery.Discovery$BlockingQueueServicesQueue.await(Discovery.java:250)
    at org.wildfly.discovery.Discovery$BlockingQueueServicesQueue.takeService(Discovery.java:282)
    at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.doAnyDiscovery(DiscoveryEJBClientInterceptor.java:493)

"http-executor task-1" #363 prio=5 os_prio=0 tid=0x00007f864550c800 nid=0x7a3 waiting on condition [0x00007f861b54b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000f910bd80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.wildfly.discovery.Discovery$BlockingQueueServicesQueue.await(Discovery.java:250)
    at org.wildfly.discovery.Discovery$BlockingQueueServicesQueue.takeService(Discovery.java:282)
    at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.doAnyDiscovery(DiscoveryEJBClientInterceptor.java:493)