org.jboss.cache.lock.TimeoutException: read lock for /_BUDDY_BACKUP during shutdown of JBoss

Solution Unverified - Updated -

Issue

We have configured a cluster with 3 nodes and have a nightly script to sequentially stop the nodes, deploy the new program version and start the server again.
The script works most of the time, but recently the script failed as one of the instances didn't shutdown.
The logs showed undeployment of the ears followed by the following errors in

Mar 26 23:40:26 localhost.localdomain org.jboss.cache.lock.TimeoutException: read lock for /_BUDDY_BACKUP_/10.20.30.103_55200/sfsb/ear=myApp.ear,jar=EJBApp1.jar,name=CheckRatings,service=EJB3 could not be acquired by Thread[AsyncViewChangeHandlerThread,10.20.30.101:55200,5,jboss] after 15000 ms. Locks: Read lock owners: [] Write lock owner: Thread[JBoss Shutdown Hook,5,jboss] , lock info: write owner=Thread[JBoss Shutdown Hook,5,jboss] (activeReaders=0, activeWriter=Thread[JBoss Shutdown Hook,5,jboss], waitingReaders=0, waitingWriters=0, waitingUpgrader=0)
     at org.jboss.cache.lock.IdentityLock.acquireReadLock0(IdentityLock.java:332)
     at org.jboss.cache.lock.IdentityLock.acquireReadLock(IdentityLock.java:276)
     at org.jboss.cache.lock.IdentityLock.acquire(IdentityLock.java:499)
     at org.jboss.cache.lock.IdentityLock.acquireAll(IdentityLock.java:555)
     at org.jboss.cache.lock.IdentityLock.acquireAll(IdentityLock.java:563)
     at org.jboss.cache.lock.IdentityLock.acquireAll(IdentityLock.java:563)

After about 7hrs undeployment of the ears continued again, followed by

WARN [org.jgroups.protocols.FC] Received two credit requests from 127.0.0.2:64842 without any intervening messages; sending 2000000 credits

but still the JBoss instance didn't shutdown

Environment

  • JBoss Enterprise Application Platform (EAP)
    • 5.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content