Many EJB client threads created during clustered remote EJB calls leads to OutOfMemoryError

Issue

• Remote calls to stateless session beans deployed in EJB cluster results in many EJB client threads created, causing OutOfMemoryError.
• Thousands of ejb-client-cluster-node-connection-creation threads.
• Thread dump analysis shows many threads waiting on a org.jboss.ejb.client.remoting.ConnectionPool monitor. For example:

ejb-client-cluster-node-connection-creation-3-thread-1234

Monitor:
Waiting for Monitor Lock on org/jboss/ejb/client/remoting/ConnectionPool@0x0000000012A34B56

Java Stack:
at org/jboss/ejb/client/remoting/RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51(Compiled Code)) 
at org/jboss/ejb/client/remoting/RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:77(Compiled Code)) 
at org/jboss/ejb/client/ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:416(Compiled Code)) 
at org/jboss/ejb/client/ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:390(Compiled Code)) 
at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:314(Compiled Code)) 
at java/util/concurrent/FutureTask.run(FutureTask.java:149(Compiled Code)) 
at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:908(Compiled Code)) 
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:931(Compiled Code)) 
at java/lang/Thread.run(Thread.java:773(Compiled Code)) 

• Thread dump analysis shows the ejb-client-cluster-node-connection-creation that owns the org.jboss.ejb.client.remoting.ConnectionPool monitor is attempting a connection:

Thread Name:
ejb-client-cluster-node-connection-creation-3-thread-6789

State:
Waiting on condition

Monitor:
Owns Monitor Lock on org/jboss/ejb/client/remoting/ConnectionPool@0x0000000012A34B56

Java Stack:
at sun/nio/ch/SelectorImpl.register(SelectorImpl.java:133(Compiled Code)) 
at java/nio/channels/spi/AbstractSelectableChannel.register(AbstractSelectableChannel.java:191(Compiled Code)) 
at java/nio/channels/SelectableChannel.register(SelectableChannel.java:265(Compiled Code)) 
at org/xnio/nio/WorkerThread.addChannel(WorkerThread.java:288(Compiled Code)) 
at org/xnio/nio/AbstractNioStreamChannel.start(AbstractNioStreamChannel.java:82(Compiled Code)) 
at org/xnio/nio/NioTcpChannel. (NioTcpChannel.java:72(Compiled Code)) 
at org/xnio/nio/NioXnioWorker.connectTcpStream(NioXnioWorker.java:313(Compiled Code)) 
at org/xnio/XnioWorker.connectStream(XnioWorker.java:179(Compiled Code)) 
at org/jboss/remoting3/remote/RemoteConnectionProvider.connect(RemoteConnectionProvider.java:214(Compiled Code)) 
at org/jboss/remoting3/EndpointImpl.doConnect(EndpointImpl.java:296(Compiled Code)) 
at org/jboss/remoting3/EndpointImpl.connect(EndpointImpl.java:386(Compiled Code)) 
at org/jboss/ejb/client/remoting/NetworkUtil.connect(NetworkUtil.java:152(Compiled Code)) 
at org/jboss/ejb/client/remoting/NetworkUtil.connect(NetworkUtil.java:133(Compiled Code)) 
at org/jboss/ejb/client/remoting/ConnectionPool.getConnection(ConnectionPool.java:75(Compiled Code)) 
at org/jboss/ejb/client/remoting/RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51(Compiled Code)) 
at org/jboss/ejb/client/remoting/RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:77(Compiled Code)) 
at org/jboss/ejb/client/ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:416(Compiled Code)) 
at org/jboss/ejb/client/ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:390(Compiled Code)) 
at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:314(Compiled Code)) 
at java/util/concurrent/FutureTask.run(FutureTask.java:149(Compiled Code)) 
at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:908(Compiled Code)) 
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:931(Compiled Code)) 
at java/lang/Thread.run(Thread.java:773(Compiled Code)) 

• A javacore is created as a result of OutOfMemoryError due to native memory exhaustion:

1TISIGINFO     Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "native memory exhausted" received