EAP 6 EJB threads bottleneck in the classloader when another nodes joins the cluster
Issue
Third JBoss node hangs when joining a cluster which is already handling requests.
Use Case
- We run 2 Jboss nodes on different machine to create a cluster.
./standalone.sh -c standalone-full-ha.xml -b eth1 -bmanagement eth1 -u 230.0.0.9 -Djboss.node.name=node1 -Djgroups.bind.addr=eth1
./standalone.sh -c standalone-full-ha.xml -b eth2 -bmanagement eth2 -u 230.0.0.9 -Djboss.node.name=node2 -Djgroups.bind.addr=10.20.30.42
-
We are using two caches, "ejb" for the cluster communication and "testConfigRepl" for report configurations needed when handling every incoming call. The config cache is replicated and uses TCP as protocol.
-
The client has been connected to the nodes using Remote Clustered EJB
- The client is sending request to the nodes with rate 800 requests per second. And both the servers are executing the requests successfully.
- Let a third node join the cluster, with a similar command:
./standalone.sh -c standalone-full-ha.xml -b eth3 -bmanagement eth3 -u 230.0.0.9 -Djboss.node.name=node3 -Djgroups.bind.addr=eth3
Observations
After the new node has declared "JBoss EAP 6.1.0.GA started ..." and then "Starting JGroups channel" (for replicated cache), it manages to handle a couple of calls. Then it hangs.
- A number threads have stacks similar to
"EJB default - 98" prio=10 tid=0x00007fab5c13c800 nid=0x3a18 in Object.wait() [0x00007fab43c48000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000077758da50> (a org.jboss.osgi.framework.internal.HostBundleClassLoader)
at java.lang.Object.wait(Object.java:503)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:415)
- A number in RUNNABLE state but waiting on condition ( they are calling different methods in the application so not all the same)
- A number of threads blocked waiting for the classloader ( but there are not hanging at the same method invocation)
EJB default - 86" prio=10 tid=0x00007fab681d6000 nid=0x3a08 waiting for monitor entry [0x00007fab44c5a000]
java.lang.Thread.State: RUNNABLE
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2413)
at java.lang.Class.getConstructor0(Class.java:2723)
at java.lang.Class.newInstance0(Class.java:345)
at java.lang.Class.newInstance(Class.java:327)
at com.example....
- ClassLoader thread stack shows the following in most of the thread dumps.
"ClassLoader Thread" daemon prio=10 tid=0x00007fab88208800 nid=0x3939 waiting on condition [0x00007fab4ef56000]
java.lang.Thread.State: RUNNABLE
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:344)
at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:422)
at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:260)
at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:75)
at org.jboss.modules.FilteredIterableLocalLoader.loadClassLocal(FilteredIterableLocalLoader.java:48)
at org.jboss.modules.Module.loadModuleClass(Module.java:526)
at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:188)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:444)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:432)
at org.jboss.modules.ConcurrentClassLoader.access$400(ConcurrentClassLoader.java:51)
at org.jboss.modules.ConcurrentClassLoader$LoaderThread.run(ConcurrentClassLoader.java:603)
- locked <0x00000007755e4fc0> (a org.jboss.osgi.framework.internal.HostBundleClassLoader)
Environment
- Red Hat JBoss Enterprise Application Platform (EAP) 6.x
- Oracle JDK 1.7 , OpenJDK 1.7
- JBoss OSGi class loader
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.