Performance slowdown and high number of "dead" class loaders in JBoss EAP 6.0.x / 6.1.x

Issue

Running JBoss EAP 6.1.0, We have a configuration where units of work (tasks) are generated on one server and then other servers configured as task workers pull tasks from the server that generated the tasks, process the tasks and publish the results. Pulling the tasks from the task generator is just done via an EJB call. We have noticed that performance severely degrades on the task workers after processing several thousand tasks.

What I am seeing in console is that the number of classes loaded continually increases. When a task worker is first brought up it will show around 14,000 classes loaded. I ran a test where I generated 240,314 and started only one task worker to process them. The number of classes loaded continued to increase to over 200,000. As the number of classes loaded increased performance decreased.

The task worker is configured with 8 threads to pull and process tasks. I performed a thread dump while processing was still being performed. All 8 of the task threads had the same stack trace as shown below. It appears that each of the threads is in the process of deserializing the task object from the remote server. In particular I noticed the Class.newInstance call about 4 lines into the stack trace.

I have also attached the output of a jmap -permstat that I performed. It shows a total of 213,484 class loaders, of which only 1 is alive. The other 213,483 are marked as dead. Most of the "dead" class loaders are sun/reflect/DelegatingClassLoader, but there are also 232 of org/jboss/modules/ModuleClassLoader. If we perform a GC then most of the classes will be unloaded and the number of loaded classes shown in console will go back down to the number at startup. Performance will go back to normal. It seems to me that the performance degradation is due to the high number of classes loaded and dead class loaders which appear to be a result of performing EJB calls. It looks like some kind of class or class loader leak.

"priority-example-pool-97-thread-8" prio=10 tid=0x00007f6a5d8fb000 nid=0x46af runnable [0x00007f6a67bee000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2483)
        at java.lang.Class.getConstructor0(Class.java:2793)
        at java.lang.Class.newInstance(Class.java:345)
        at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399)
        at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395)
        at sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenerator.java:113)
        at sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:331)
        at org.jboss.marshalling.reflect.SerializableClass$NoInitConstructorFinder.get(SerializableClass.java:768)
        at org.jboss.marshalling.reflect.SerializableClass$LazyWeakConstructorRef.getConstructor(SerializableClass.java:676)
        at org.jboss.marshalling.reflect.SerializableClass$LazyWeakConstructorRef.access$300(SerializableClass.java:636)
        at org.jboss.marshalling.reflect.SerializableClass.getNoInitConstructor(SerializableClass.java:563)
        at org.jboss.marshalling.reflect.SunReflectiveCreator.getNewConstructor(SunReflectiveCreator.java:38)
        at org.jboss.marshalling.reflect.ReflectiveCreator.create(ReflectiveCreator.java:71)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1233)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
        at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
        at org.jboss.ejb.client.remoting.MethodInvocationResponseHandler$MethodInvocationResultProducer.getResult(MethodInvocationResponseHandler.java:107)
        at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:272)
        at org.jboss.ejb.client.EJBObjectInterceptor.handleInvocationResult(EJBObjectInterceptor.java:64)
        at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:274)
        at org.jboss.ejb.client.EJBHomeInterceptor.handleInvocationResult(EJBHomeInterceptor.java:88)
        at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:274)
        at org.jboss.ejb.client.TransactionInterceptor.handleInvocationResult(TransactionInterceptor.java:46)
        at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:274)
        at org.jboss.ejb.client.ReceiverInterceptor.handleInvocationResult(ReceiverInterceptor.java:129)
        at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:262)
        at org.jboss.ejb.client.EJBClientInvocationContext.awaitResponse(EJBClientInvocationContext.java:437)
        at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:202)
        at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:181)
        at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:144)
        at com.sun.proxy.$Proxy96.receiveTaskMessage(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        ...