JBoss EJB calls are stalling in the EntityBeanSynchronizationInterceptor

Solution Unverified - Updated -

Environment

  • JBoss Enterprise Application Platform (EAP) 6

Issue

  • JBoss is stalling with many EJB calls blocked like so in EntityBeanSynchronizationInterceptor:
"http-/0.0.0.0:8080-1" daemon prio=10 tid=0x000000000f0f2000 nid=0x3935 waiting on condition [0x000000004ee6a000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007a6a9b248> (a org.jboss.as.ejb3.tx.OwnableReentrantLock)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
    at org.jboss.as.ejb3.tx.OwnableReentrantLock.lock(OwnableReentrantLock.java:86)
    at org.jboss.as.ejb3.component.entity.interceptors.EntityBeanSynchronizationInterceptor.processInvocation(EntityBeanSynchronizationInterceptor.java:80)

Resolution

  • Use optimistic locking instead, which will allow multiple instances of an entity bean to be active simultaneously. Enable it via CLI:
/subsystem=ejb3:write-attribute(name=default-entity-bean-optimistic-locking,value=true)
  • Or directly in your xml:
        <subsystem xmlns="urn:jboss:domain:ejb3:1.3">
            ...
            <entity-bean>
                <optimistic-locking enabled="true"/>
            </entity-bean>
        </subsystem>

Root Cause

  • Deadlocks are occurring across several EntityBeanSynchronizationInterceptor locks since entity beans are using pessimistic locking.
  • Pessimistic locking allows only one transaction active in a entity bean so if many transactions involve multiple entity beans and the same entity beans, then they could deadlock trying to lock pessimistically on the entity beans until a transaction timeout breaks them out.

Diagnostic Steps

  • Troubleshoot using thread dumps and additional steps mentioned in Java application unresponsive
  • Since ownership of these locks isn't directly exposed via thread dumps, enable EJB3 TRACE logging to further track ownership of these locks:
        <subsystem xmlns="urn:jboss:domain:logging:1.1">
            ...
            <logger category="org.jboss.as.ejb3">
                <level name="TRACE"/>
            </logger>
            <root-logger>
                <level name="INFO"/>
                <handlers>
                    <handler name="FILE"/>
                </handlers>
            </root-logger>
        </subsystem>
  • Once you've identified the stalled thread(s) in the thread dumps, see what lock it is waiting for by checking its last logged "trying to acquire lock" TRACE message like the following:
TRACE (http-/0.0.0.0:8080-1) [org.jboss.as.ejb3] Trying to acquire lock: org.jboss.as.ejb3.tx.OwnableReentrantLock@2c9cf5b[State = 1, nonempty queue][Locked by 0:ffff0a5c146c:65ebda30:51140fb9:532d53] for entity bean org.jboss.as.cmp.component.CmpEntityBeanComponentInstance@f4c60cb during invocation: org.jboss.invocation.InterceptorContext@12d2f69b
  • Then identify the thread that owns this lock by checking for the last logged "acquired" message for the OwnableReentrantLock previously identified:
TRACE (http-/0.0.0.0:8080-2) [org.jboss.as.ejb3] Acquired lock: org.jboss.as.ejb3.tx.OwnableReentrantLock@2c9cf5b[State = 2, nonempty queue][Locked by 0:ffff0a5c146c:65ebda30:51140fb9:532d53] for entity bean instance: org.jboss.as.cmp.component.CmpEntityBeanComponentInstance@f4c60cb during invocation: org.jboss.invocation.InterceptorContext@6d1b39a8
  • Further review thread dumps and logging to determine what that lock owning thread is doing. You may see that it is deadlocked in some manner with the thread trying to acquire its OwnableReentrantLock.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.