Deadlock on JBoss startup

Latest response

Hi all,

I am having some issues where from time to time my JBoss 5.2 instance(s) won't fully start up.

I can see in the logs, that the JBoss instance(s) gets until the ** INFO [Version] HV000001: Hibernate Validator 4.3.1.Final ** point, and after that it just hangs and never completes. I have found so far that this happens due to deadlocks that occur while the instance(s) is starting. Running "jstack
", would show me the deadlock which i provide below. The instances are running an ATG10.2 application. I don't see any resource issues on my servers(plenty of memory, plenty of space)

Here is where the deadlock happens:

Found one Java-level deadlock:

"Thread-12":
waiting to lock monitor 0x000000004956e470 (object 0x0000000400d9e870, a java.lang.Object),
which is held by "main"
"main":
waiting for ownable synchronizer 0x00000004013f1bd8, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by "Thread-12"

And here are the top line of the stack for "main" and "Thread-12":

Java stack information for the threads listed above:

"Thread-12":
at java.beans.Introspector.getBeanInfo(Introspector.java:157)
- waiting to lock <0x0000000400d9e870> (a java.lang.Object)
at org.jboss.aop.advice.GenericAspectFactory.configureInstance(GenericAspectFactory.java:223)
at org.jboss.aop.advice.GenericAspectFactory.createPerVM(GenericAspectFactory.java:147)
at org.jboss.aop.AspectManager.createPerVmAspect(AspectManager.java:2052)
- locked <0x00000006d7b9c0b0> (a org.jboss.aop.advice.AspectDefinition)
at org.jboss.aop.AspectManager.getPerVMAspect(AspectManager.java:2028)
- locked <0x00000006d7b9c0b0> (a org.jboss.aop.advice.AspectDefinition)
at org.jboss.aop.AspectManager.getPerVMAspect(AspectManager.java:2011)
at org.jboss.aop.advice.PerVmAdvice.generateOptimized(PerVmAdvice.java:51)
- locked <0x00000003f4949dc0> (a java.lang.Class for org.jboss.aop.advice.PerVmAdvice)
at org.jboss.aop.advice.AdviceFactory.create(AdviceFactory.java:106)
at org.jboss.aop.Advisor.createInterceptorChain(Advisor.java:1328)
at org.jboss.aop.Advisor.pointcutResolved(Advisor.java:1942)
at org.jboss.aop.MethodMatchInfo.applyBinding(MethodMatchInfo.java:152)
at org.jboss.aop.MethodMatchInfo.simplePopulateBindings(MethodMatchInfo.java:113)
at org.jboss.aop.MethodMatchInfo.populateBindings(MethodMatchInfo.java:96)
at org.jboss.aop.Advisor.finalizeMethodChain(Advisor.java:1553)
at org.jboss.aop.ClassAdvisor.finalizeChains(ClassAdvisor.java:741)
at org.jboss.aop.ClassAdvisor.doRebuildInterceptorsForAddedBinding(ClassAdvisor.java:1163)
at org.jboss.aop.ClassAdvisor$RebuildInterceptorsAction$2.rebuildInterceptorsForAddedBinding(ClassAdvisor.java:2367)
at org.jboss.aop.ClassAdvisor.rebuildInterceptorsForAddedBinding(ClassAdvisor.java:1141)
at org.jboss.aop.Advisor.newBindingAdded(Advisor.java:1134)
- locked <0x00000007487bcde0> (a org.jboss.aop.ClassAdvisor)
at org.jboss.aop.AspectManager.updateAdvisorsForAddedBinding(AspectManager.java:1530)
- locked <0x00000004012d09b0> (a java.util.WeakHashMap)
at org.jboss.aop.AspectManager.addBinding(AspectManager.java:1491)
- locked <0x00000004012d09b0> (a java.util.WeakHashMap)
at org.jboss.aop.AspectXmlLoader.deployBinding(AspectXmlLoader.java:292)
at org.jboss.aop.AspectXmlLoader.deployTopElements(AspectXmlLoader.java:1145)
at org.jboss.aop.AspectXmlLoader.deployXML(AspectXmlLoader.java:990)
at org.jboss.aop.AspectXmlLoader.deployXML(AspectXmlLoader.java:981)
at org.jboss.jms.client.container.JmsClientAspectXMLLoader.deployXML(JmsClientAspectXMLLoader.java:93)
- locked <0x00000004012d0908> (a org.jboss.aop.AspectManager)

"main":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004013f1bd8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:677)
at org.jboss.aop.util.AOPLock.lockRead(AOPLock.java:40)
at org.jboss.aop.AspectManager.translate(AspectManager.java:1060)
at org.jboss.aop.AspectManager.transform(AspectManager.java:1023)
at org.jboss.aop.standalone.AOPTransformer.aspectTransform(AOPTransformer.java:87)
at org.jboss.aop.standalone.AOPTransformer.transform(AOPTransformer.java:75)
at sun.instrument.TransformerManager.transform(TransformerManager.java:169)
at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:365)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:637)
at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
at org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:48)
at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:637)
at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:596)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:595)
- locked <0x00000007406c4210> (a org.jboss.classloader.spi.base.BaseClassLoader)
at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:572)
at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:135)
at org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:455)
at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:267)
at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:166)
at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:304)
at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1261)
at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:914)
at org.jboss.classloader.spi.base.BaseClassLoader.doLoadClass(BaseClassLoader.java:506)
- locked <0x00000007406c4210> (a org.jboss.classloader.spi.base.BaseClassLoader)

Anyone has ran into this deadlock before? Any suggestion is highly appreciated. If you need any other information, please let me know.
If it helps to see the full stack of the two deadlocks, i can also provide that.

Thank you in advance !

Attachments

Responses

What Java version is this running under. We're probably going to need to see the full thread stack and such so you may be better of raising a case against this issue. First of all take a look a the following Knowledge Base Articles to see if they provide any further assistance to you [1] and [2]

[1] https://access.redhat.com/solutions/23742
[2] https://access.redhat.com/solutions/65859

Hi Mustafa,
Thank you for your reply. I am running:

java version "1.6.0_75"
Java(TM) SE Runtime Environment (build 1.6.0_75-b13)
Java HotSpot(TM) 64-Bit Server VM (build 20.75-b01, mixed mode)

Runnin "/usr/sbin/alternatives --config java", i can see that the selected jdk is: /usr/lib/jvm/jre-1.6.0-sun.x86_64/bin/java

I have uploaded also the full stack of the two deadlocked threads. As I am reading the first link. As for the second link you posted, the solution it suggests does not apply in my case, as i already a running JBoss 5.2. The second link suggests to upgrade to 5.1.x

Thank you !

Hi Pierin

If you need urgent assistance with this then I'd suggest you raise a case. Otherwise I'll try and get one of our JVM engineers to take a look at your stack trace as soon as possible.

Thanks
Mustafa

Hi Mustafa,

Please, try to see if you can get one of your engineers to take a look at the stack trace. If it becomes urgent and we still have not seen any solution in this discussion or elsewhere, then will raise a case.

Cheers,
Pierin

Hi Mustafa
Have updated the uploaded stack trace file, as now contains the entire output of "jstack PID"

Thank you !

Thanks Pierin. Usually if an issue is complex enough to require analysis of a stack trace we would encourage you to open a support case for it. Hopefully someone will be able to assist here, but please don't hesitate to open a case if required.

Was there ever a resolution to this issue?

I believe the issue was not yet fully resolved, but Pierin may be able to suggest a workaround.

Hello Ed, David,

I eventually opened a support ticket with RedHat for this issue. One of their suggestions was to upgrade to Java 7, but currently we can't do that cause our application does not entirely support it.

We still have deadlocks appearing during the start up of the instances, but because of other tasks and projects in our portfolio, as a workaround we restart the instance(s) whenever they start up with a deadlock. We will get back to this issue with more focus when the time allows it.
So, a complete resolution has not been found yet.

Thank you for your time and interest,
Pierin

Hi Ed,

I forgot to ask. Are you having a similar issue ?

Thanks

Hi Pierin,

I ran into the issue working with the JON server. Different version of JBoss and I'm running Java 7, but same error. It appears to be intermittent though, as you've found.