EAP 6 EJB threads bottleneck in the classloader when another nodes joins the cluster

Solution Verified - Updated -

Issue

Third JBoss node hangs when joining a cluster which is already handling requests.

Use Case

  • We run 2 Jboss nodes on different machine to create a cluster.
./standalone.sh -c standalone-full-ha.xml -b eth1 -bmanagement eth1 -u 230.0.0.9 -Djboss.node.name=node1 -Djgroups.bind.addr=eth1
./standalone.sh -c standalone-full-ha.xml -b eth2 -bmanagement eth2 -u 230.0.0.9 -Djboss.node.name=node2 -Djgroups.bind.addr=10.20.30.42
  • We are using two caches, "ejb" for the cluster communication and "testConfigRepl" for report configurations needed when handling every incoming call. The config cache is replicated and uses TCP as protocol.

  • The client has been connected to the nodes using Remote Clustered EJB

  • The client is sending request to the nodes with rate 800 requests per second. And both the servers are executing the requests successfully.
  • Let a third node join the cluster, with a similar command:
    ./standalone.sh -c standalone-full-ha.xml -b eth3 -bmanagement eth3 -u 230.0.0.9 -Djboss.node.name=node3 -Djgroups.bind.addr=eth3

Observations
After the new node has declared "JBoss EAP 6.1.0.GA started ..." and then "Starting JGroups channel" (for replicated cache), it manages to handle a couple of calls. Then it hangs.
- A number threads have stacks similar to

"EJB default - 98" prio=10 tid=0x00007fab5c13c800 nid=0x3a18 in Object.wait() [0x00007fab43c48000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x000000077758da50> (a org.jboss.osgi.framework.internal.HostBundleClassLoader)
    at java.lang.Object.wait(Object.java:503)
    at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:415)
  • A number in RUNNABLE state but waiting on condition ( they are calling different methods in the application so not all the same)
  • A number of threads blocked waiting for the classloader ( but there are not hanging at the same method invocation)
EJB default - 86"   prio=10 tid=0x00007fab681d6000 nid=0x3a08 waiting for monitor entry [0x00007fab44c5a000]
     java.lang.Thread.State: RUNNABLE
      at java.lang.Class.getDeclaredConstructors0(Native Method)
      at java.lang.Class.privateGetDeclaredConstructors(Class.java:2413)
      at java.lang.Class.getConstructor0(Class.java:2723)
      at java.lang.Class.newInstance0(Class.java:345)
      at java.lang.Class.newInstance(Class.java:327)
      at com.example....
  • ClassLoader thread stack shows the following in most of the thread dumps.
"ClassLoader Thread" daemon prio=10 tid=0x00007fab88208800 nid=0x3939 waiting on condition [0x00007fab4ef56000]
   java.lang.Thread.State: RUNNABLE
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
    at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:344)
    at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:422)
    at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:260)
    at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:75)
    at org.jboss.modules.FilteredIterableLocalLoader.loadClassLocal(FilteredIterableLocalLoader.java:48)
    at org.jboss.modules.Module.loadModuleClass(Module.java:526)
    at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:188)
    at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:444)
    at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:432)
    at org.jboss.modules.ConcurrentClassLoader.access$400(ConcurrentClassLoader.java:51)
    at org.jboss.modules.ConcurrentClassLoader$LoaderThread.run(ConcurrentClassLoader.java:603)
    - locked <0x00000007755e4fc0> (a org.jboss.osgi.framework.internal.HostBundleClassLoader)

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP) 6.x
  • Oracle JDK 1.7 , OpenJDK 1.7
  • JBoss OSGi class loader

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content