Many EJB client threads created during clustered remote EJB calls leads to OutOfMemoryError

Solution Unverified - Updated -

Issue

  • Remote calls to stateless session beans deployed in EJB cluster results in many EJB client threads created, causing OutOfMemoryError.
  • Thousands of ejb-client-cluster-node-connection-creation threads.
  • Thread dump analysis shows many threads waiting on a org.jboss.ejb.client.remoting.ConnectionPool monitor. For example:
ejb-client-cluster-node-connection-creation-3-thread-1234

Monitor:
Waiting for Monitor Lock on org/jboss/ejb/client/remoting/ConnectionPool@0x0000000012A34B56

Java Stack:
at org/jboss/ejb/client/remoting/RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51(Compiled Code)) 
at org/jboss/ejb/client/remoting/RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:77(Compiled Code)) 
at org/jboss/ejb/client/ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:416(Compiled Code)) 
at org/jboss/ejb/client/ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:390(Compiled Code)) 
at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:314(Compiled Code)) 
at java/util/concurrent/FutureTask.run(FutureTask.java:149(Compiled Code)) 
at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:908(Compiled Code)) 
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:931(Compiled Code)) 
at java/lang/Thread.run(Thread.java:773(Compiled Code)) 
  • Thread dump analysis shows the ejb-client-cluster-node-connection-creation that owns the org.jboss.ejb.client.remoting.ConnectionPool monitor is attempting a connection:
Thread Name:
ejb-client-cluster-node-connection-creation-3-thread-6789

State:
Waiting on condition

Monitor:
Owns Monitor Lock on org/jboss/ejb/client/remoting/ConnectionPool@0x0000000012A34B56

Java Stack:
at sun/nio/ch/SelectorImpl.register(SelectorImpl.java:133(Compiled Code)) 
at java/nio/channels/spi/AbstractSelectableChannel.register(AbstractSelectableChannel.java:191(Compiled Code)) 
at java/nio/channels/SelectableChannel.register(SelectableChannel.java:265(Compiled Code)) 
at org/xnio/nio/WorkerThread.addChannel(WorkerThread.java:288(Compiled Code)) 
at org/xnio/nio/AbstractNioStreamChannel.start(AbstractNioStreamChannel.java:82(Compiled Code)) 
at org/xnio/nio/NioTcpChannel. (NioTcpChannel.java:72(Compiled Code)) 
at org/xnio/nio/NioXnioWorker.connectTcpStream(NioXnioWorker.java:313(Compiled Code)) 
at org/xnio/XnioWorker.connectStream(XnioWorker.java:179(Compiled Code)) 
at org/jboss/remoting3/remote/RemoteConnectionProvider.connect(RemoteConnectionProvider.java:214(Compiled Code)) 
at org/jboss/remoting3/EndpointImpl.doConnect(EndpointImpl.java:296(Compiled Code)) 
at org/jboss/remoting3/EndpointImpl.connect(EndpointImpl.java:386(Compiled Code)) 
at org/jboss/ejb/client/remoting/NetworkUtil.connect(NetworkUtil.java:152(Compiled Code)) 
at org/jboss/ejb/client/remoting/NetworkUtil.connect(NetworkUtil.java:133(Compiled Code)) 
at org/jboss/ejb/client/remoting/ConnectionPool.getConnection(ConnectionPool.java:75(Compiled Code)) 
at org/jboss/ejb/client/remoting/RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51(Compiled Code)) 
at org/jboss/ejb/client/remoting/RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:77(Compiled Code)) 
at org/jboss/ejb/client/ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:416(Compiled Code)) 
at org/jboss/ejb/client/ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:390(Compiled Code)) 
at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:314(Compiled Code)) 
at java/util/concurrent/FutureTask.run(FutureTask.java:149(Compiled Code)) 
at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:908(Compiled Code)) 
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:931(Compiled Code)) 
at java/lang/Thread.run(Thread.java:773(Compiled Code)) 
  • A javacore is created as a result of OutOfMemoryError due to native memory exhaustion:
1TISIGINFO     Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "native memory exhausted" received

Environment

  • Websphere EJB client
  • JBoss Enterprise Application Platform (EAP) backend
    • 6.3.1
  • Scoped EJB client context

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.