EAP 6 EJB client deadlock

Issue

• Occassionally, we see a deadlock like the following in EJB client code between org.jboss.ejb.client.ClusterContext.receiverContextClosed and org.jboss.ejb.client.EJBClientContext.getNodeEJBReceiver :

"Remoting "config-based-ejb-client-endpoint" read-1":
    at java.util.Collections$SynchronizedCollection.remove(Collections.java:1580)
    - waiting to lock <0x000000078609c840> (a java.util.Collections$SynchronizedSet)
    at org.jboss.ejb.client.ClusterContext.receiverContextClosed(ClusterContext.java:414)
    at org.jboss.ejb.client.EJBClientContext.unregisterEJBReceiver(EJBClientContext.java:443)
    - locked <0x000000078608c238> (a java.util.IdentityHashMap)
    at org.jboss.ejb.client.EJBReceiverContext.close(EJBReceiverContext.java:59)
    at org.jboss.ejb.client.remoting.ChannelAssociation.notifyBrokenChannel(ChannelAssociation.java:407)
    at org.jboss.ejb.client.remoting.ChannelAssociation.access$100(ChannelAssociation.java:59)
    at org.jboss.ejb.client.remoting.ChannelAssociation$1.handleClose(ChannelAssociation.java:118)
    at org.jboss.ejb.client.remoting.ChannelAssociation$1.handleClose(ChannelAssociation.java:110)
    at org.jboss.remoting3.spi.SpiUtils.safeHandleClose(SpiUtils.java:54)
    at org.jboss.remoting3.spi.AbstractHandleableCloseable$CloseHandlerTask.run(AbstractHandleableCloseable.java:514)
    at org.jboss.remoting3.spi.AbstractHandleableCloseable.runCloseTask(AbstractHandleableCloseable.java:419)
    at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeComplete(AbstractHandleableCloseable.java:290)
    at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:531)
    at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:372)
    at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:437)
    at org.jboss.remoting3.remote.RemoteConnectionHandler.receiveCloseRequest(RemoteConnectionHandler.java:217)
    at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:113)
    at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:46)
    at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
    at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
    at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
    at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
    at org.xnio.nio.NioHandle.run(NioHandle.java:90)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:198)

"Client09":
    at org.jboss.ejb.client.EJBClientContext.getNodeEJBReceiver(EJBClientContext.java:860)
    - waiting to lock <0x000000078608c238> (a java.util.IdentityHashMap)
    at org.jboss.ejb.client.EJBClientContext.getNodeEJBReceiver(EJBClientContext.java:846)
    at org.jboss.ejb.client.EJBClientContext.getNodeEJBReceiverContext(EJBClientContext.java:888)
    at org.jboss.ejb.client.ClusterContext.isNodeConnectedAndDeployed(ClusterContext.java:251)
    at org.jboss.ejb.client.ClusterContext.getConnectedAndDeployedNodes(ClusterContext.java:266)
    - locked <0x000000078609c840> (a java.util.Collections$SynchronizedSet)
    at org.jboss.ejb.client.ClusterContext.getEJBReceiverContext(ClusterContext.java:125)
    at org.jboss.ejb.client.ClusterContext.getEJBReceiverContext(ClusterContext.java:96)
    at org.jboss.ejb.client.EJBClientContext.getClusterEJBReceiverContext(EJBClientContext.java:958)
    at org.jboss.ejb.client.EJBClientContext.getClusterEJBReceiverContext(EJBClientContext.java:938)
    at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:108)
    at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
    at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:255)
    at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:200)
    at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:183)
    at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)