org.jboss.cache.lock.TimeoutException: read lock for /_BUDDY_BACKUP during shutdown of JBoss
Issue
We have configured a cluster with 3 nodes and have a nightly script to sequentially stop the nodes, deploy the new program version and start the server again.
The script works most of the time, but recently the script failed as one of the instances didn't shutdown.
The logs showed undeployment of the ears followed by the following errors in
Mar 26 23:40:26 localhost.localdomain org.jboss.cache.lock.TimeoutException: read lock for /_BUDDY_BACKUP_/10.20.30.103_55200/sfsb/ear=myApp.ear,jar=EJBApp1.jar,name=CheckRatings,service=EJB3 could not be acquired by Thread[AsyncViewChangeHandlerThread,10.20.30.101:55200,5,jboss] after 15000 ms. Locks: Read lock owners: [] Write lock owner: Thread[JBoss Shutdown Hook,5,jboss] , lock info: write owner=Thread[JBoss Shutdown Hook,5,jboss] (activeReaders=0, activeWriter=Thread[JBoss Shutdown Hook,5,jboss], waitingReaders=0, waitingWriters=0, waitingUpgrader=0)
at org.jboss.cache.lock.IdentityLock.acquireReadLock0(IdentityLock.java:332)
at org.jboss.cache.lock.IdentityLock.acquireReadLock(IdentityLock.java:276)
at org.jboss.cache.lock.IdentityLock.acquire(IdentityLock.java:499)
at org.jboss.cache.lock.IdentityLock.acquireAll(IdentityLock.java:555)
at org.jboss.cache.lock.IdentityLock.acquireAll(IdentityLock.java:563)
at org.jboss.cache.lock.IdentityLock.acquireAll(IdentityLock.java:563)
After about 7hrs undeployment of the ears continued again, followed by
WARN [org.jgroups.protocols.FC] Received two credit requests from 127.0.0.2:64842 without any intervening messages; sending 2000000 credits
but still the JBoss instance didn't shutdown
Environment
- JBoss Enterprise Application Platform (EAP)
- 5.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.