Remote EJB client hangs with high CPU after several hours and large number of invocations in EAP 7

Solution Verified - Updated -

Issue

  • The server seem to stop processing EJB requests. When running a load test against a JBoss EAP 7.2 CP7 or JBoss EAP 7.3 CP1 server with stateful EJB calls from a Java client, after a some time, the server seem to stop responding to requests. On the client side there is no error, but on the client side is blocked with CPU at 100%. It does not matter if the client is a standalone Java client or if the Java client is located in another JBoss EAP server.
  • Migrated from EAP 6.4 to 7.2 and under production load system becomes unresponsive.

  • Thread dump shows:

    "default-threads - 47" #1718 prio=5 os_prio=0 tid=0x000000000f786800 nid=0x3f4e runnable [0x00007f4c9a094000]
        java.lang.Thread.State: RUNNABLE
            at org.jboss.remoting3._private.IntIndexHashMap.doGet(IntIndexHashMap.java:521)
            at org.jboss.remoting3._private.IntIndexHashMap.containsKey(IntIndexHashMap.java:178)
            at org.jboss.remoting3.util.InvocationTracker.addInvocation(InvocationTracker.java:122)
            at org.jboss.ejb.protocol.remote.EJBClientChannel.processInvocation(EJBClientChannel.java:332)
            at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.lambda$handleDone$0(RemoteEJBReceiver.java:93)
            at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1$$Lambda$1556/1096920767.notify(Unknown Source)
            at org.xnio.FinishedIoFuture.addNotifier(FinishedIoFuture.java:79)
            at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.handleDone(RemoteEJBReceiver.java:78)
            at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.handleDone(RemoteEJBReceiver.java:76)
            at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208)
            at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720)
            at org.xnio.IoUtils$2.execute(IoUtils.java:71)
            at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:693)
            at org.xnio.AbstractIoFuture$CompleteState.withNotifier(AbstractIoFuture.java:132)
            at org.xnio.AbstractIoFuture.addNotifier(AbstractIoFuture.java:570)
            at org.jboss.ejb.protocol.remote.RemoteEJBReceiver.processInvocation(RemoteEJBReceiver.java:134)
    

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 7.2 CP8
    • 7.3 CP1

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In