Many EJB client threads created during clustered remote EJB calls leads to OutOfMemoryError
Issue
- Remote calls to stateless session beans deployed in EJB cluster results in many EJB client threads created, causing
OutOfMemoryError. - Thousands of
ejb-client-cluster-node-connection-creationthreads. - Thread dump analysis shows many threads waiting on a
org.jboss.ejb.client.remoting.ConnectionPoolmonitor. For example:
ejb-client-cluster-node-connection-creation-3-thread-1234
Monitor:
Waiting for Monitor Lock on org/jboss/ejb/client/remoting/ConnectionPool@0x0000000012A34B56
Java Stack:
at org/jboss/ejb/client/remoting/RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51(Compiled Code))
at org/jboss/ejb/client/remoting/RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:77(Compiled Code))
at org/jboss/ejb/client/ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:416(Compiled Code))
at org/jboss/ejb/client/ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:390(Compiled Code))
at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:314(Compiled Code))
at java/util/concurrent/FutureTask.run(FutureTask.java:149(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:908(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:931(Compiled Code))
at java/lang/Thread.run(Thread.java:773(Compiled Code))
- Thread dump analysis shows the
ejb-client-cluster-node-connection-creationthat owns theorg.jboss.ejb.client.remoting.ConnectionPoolmonitor is attempting a connection:
Thread Name:
ejb-client-cluster-node-connection-creation-3-thread-6789
State:
Waiting on condition
Monitor:
Owns Monitor Lock on org/jboss/ejb/client/remoting/ConnectionPool@0x0000000012A34B56
Java Stack:
at sun/nio/ch/SelectorImpl.register(SelectorImpl.java:133(Compiled Code))
at java/nio/channels/spi/AbstractSelectableChannel.register(AbstractSelectableChannel.java:191(Compiled Code))
at java/nio/channels/SelectableChannel.register(SelectableChannel.java:265(Compiled Code))
at org/xnio/nio/WorkerThread.addChannel(WorkerThread.java:288(Compiled Code))
at org/xnio/nio/AbstractNioStreamChannel.start(AbstractNioStreamChannel.java:82(Compiled Code))
at org/xnio/nio/NioTcpChannel. (NioTcpChannel.java:72(Compiled Code))
at org/xnio/nio/NioXnioWorker.connectTcpStream(NioXnioWorker.java:313(Compiled Code))
at org/xnio/XnioWorker.connectStream(XnioWorker.java:179(Compiled Code))
at org/jboss/remoting3/remote/RemoteConnectionProvider.connect(RemoteConnectionProvider.java:214(Compiled Code))
at org/jboss/remoting3/EndpointImpl.doConnect(EndpointImpl.java:296(Compiled Code))
at org/jboss/remoting3/EndpointImpl.connect(EndpointImpl.java:386(Compiled Code))
at org/jboss/ejb/client/remoting/NetworkUtil.connect(NetworkUtil.java:152(Compiled Code))
at org/jboss/ejb/client/remoting/NetworkUtil.connect(NetworkUtil.java:133(Compiled Code))
at org/jboss/ejb/client/remoting/ConnectionPool.getConnection(ConnectionPool.java:75(Compiled Code))
at org/jboss/ejb/client/remoting/RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51(Compiled Code))
at org/jboss/ejb/client/remoting/RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:77(Compiled Code))
at org/jboss/ejb/client/ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:416(Compiled Code))
at org/jboss/ejb/client/ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:390(Compiled Code))
at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:314(Compiled Code))
at java/util/concurrent/FutureTask.run(FutureTask.java:149(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:908(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:931(Compiled Code))
at java/lang/Thread.run(Thread.java:773(Compiled Code))
- A
javacoreis created as a result of OutOfMemoryError due to native memory exhaustion:
1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "native memory exhausted" received
Environment
- Websphere EJB client
- JBoss Enterprise Application Platform (EAP) backend
- 6.3.1
- Scoped EJB client context
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
