"org.jboss.cache.lock.TimeoutException: Unable to acquire lock on Fqn" in JBoss EAP log

Solution Unverified - Updated -

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 5.x
  • Red Hat JBoss Enterprise SOA Platform (SOA)
    • 5.x

Issue

  • 2nd level cache: TimeoutException Unable to acquire lock on Fqn

  • The following error is logged on a remote cluster member when writing to JBoss Cache

    ERROR [org.jboss.cache.interceptors.TxInterceptor] (Incoming-123,MyPartition,10.0.0.2:7800) prepare failed!
org.jboss.cache.lock.TimeoutException: Unable to acquire lock on Fqn [/foo/bar] after [15000] milliseconds for requestor [GlobalTransaction:<10.0.0.1:7800>:2222]! Lock held by [GlobalTransaction:<10.0.0.1:7800>:1111]
at org.jboss.cache.mvcc.MVCCNodeHelper.acquireLock(MVCCNodeHelper.java:157)
    ...

Resolution

Structure the cache to minimize concurrent access to the same node, particularly from threads on different cluster members.

Some bugs in JBoss Cache prior to JBoss EAP 5.2 and JBoss SOA 5.3.1, could also cause this issue in certain situations even when the application did not have concurrent access to the same cache node.
https://issues.jboss.org/browse/JBPAPP-9220
https://issues.jboss.org/browse/JBPAPP-9604
https://issues.jboss.org/browse/JBPAPP-9613
https://issues.jboss.org/browse/JBPAPP-9639

Root Cause

Another thread is already holding the write lock on the listed node, preventing the current thread from completing its transaction.

If the threads are waiting on locks the other thread holds, they will deadlock until one of them times out (in which case it will get this exception, and the other thread may complete).

Since JBoss Cache locks only the local cluster member except when committing the transaction, threads on different nodes writing to the same cache node simultaneously can cause this deadlock.

Diagnostic Steps

Log every time a node lock is acquired and released:

    <category name="org.jboss.cache.lock.MVCCLockManager">
        <priority value="TRACE"/>
    </category>

Log the transaction, so "GlobalTransaction:<10.0.0.1>:123" can be associated with a thread:

    <category name="org.jboss.cache.transaction.TransactionTable">
        <priority value="TRACE"/>
    </category>

Log the cache operation being called:

    <category name="org.jboss.cache.interceptors.InvocationContextInterceptor">
        <priority value="TRACE"/>
    </category>

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments