Threads deadlock on trying to reestablish EJB Connection in JBoss EAP 6

Issue

• Threads deadlock on trying to reestablish EJB Connection. We are getting thread deadlocks when an attempt is being made to re-establish an EJB connect. For example: lots of threads are showing up as follows:

"EJB default - 249" prio=10 tid=0x00007fe3c40a7800 nid=0xe09 waiting for monitor entry [0x00007fe33a15e000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.jboss.ejb.client.EJBClientContext.attemptReconnections(EJBClientContext.java:1098)
        - waiting to lock <0x00007fe6190a27e8> (a org.jboss.ejb.client.EJBClientContext)
        at org.jboss.ejb.client.EJBClientContext.getClusterEJBReceiverContext(EJBClientContext.java:934)
        at org.jboss.ejb.client.EJBClientContext.getClusterEJBReceiverContext(EJBClientContext.java:915)
        at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:108)
        at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:183)
        at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:253)
        at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:198)
        at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:181)
        at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:144)
        at com.sun.proxy.$Proxy83.getRecord2Rules(Unknown Source)

• When searched for the lock object 0x00007fe6190a27e8, found that its being held by another thread, which in turn is blocked waiting on a lock. See below:

"EJB default - 200" prio=10 tid=0x00007fe3bc09f000 nid=0xdd8 waiting on condition [0x00007fe33d28f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fe5d5e42020> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1033)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:282)
        at org.jboss.ejb.client.EJBClientContext.attemptReconnections(EJBClientContext.java:1122)
        - locked <0x00007fe6190a27e8> (a org.jboss.ejb.client.EJBClientContext)

• Many client reconnect-threads after cluster restart are blocked. After we restarted our server-cluster we had 3 frontends with a massive number of threads, apparently all blocked on the ReconnectHandler. We then had to restart these clients. The thread dump showed the similar stacktrace as mentioned above.