Slow/Blocked EJB client when multiple JBoss EAP instances are configured as backedn EJB servers in unstable network environment

Solution Verified - Updated -

Issue

When EJB client (either standalone Java EJB client, or another JBoss EAP server in server-to-server EJB calls) is configured to connect to multiple back-end EJB/EAP server instances, it is possible to see EJB client becomes slow or blocked, when one (or multiple) backend EJB node(s) drops off network, or has slow network connections.

Capturing thread dumps when EJB client becomes slow or blocked, we may see similar threads as below waiting on DiscoveryEJBClientInterceptor.doAnyDiscovery():

"http-executor task-3" #801 prio=5 os_prio=0 tid=0x0000556d51610800 nid=0xca8 waiting on condition [0x00007f8617b4b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000dc522618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.wildfly.discovery.Discovery$BlockingQueueServicesQueue.await(Discovery.java:250)
    at org.wildfly.discovery.Discovery$BlockingQueueServicesQueue.takeService(Discovery.java:282)
    at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.doAnyDiscovery(DiscoveryEJBClientInterceptor.java:493)

"http-executor task-1" #363 prio=5 os_prio=0 tid=0x00007f864550c800 nid=0x7a3 waiting on condition [0x00007f861b54b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000f910bd80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.wildfly.discovery.Discovery$BlockingQueueServicesQueue.await(Discovery.java:250)
    at org.wildfly.discovery.Discovery$BlockingQueueServicesQueue.takeService(Discovery.java:282)
    at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.doAnyDiscovery(DiscoveryEJBClientInterceptor.java:493)

Environment

  • Red Hat JBoss Enterprise Application Platform (JBoss EAP)
    • 7.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In