Hot Rod client and JDG servers get stuck for about 30 seconds during shutdown of one cluster node in JDG6

Solution Verified - Updated -

Issue

  • If a JDG server cluster is under load and one of the servers is about to shut down it is expected that all clients and other servers are continue to work and not being affected. But sometimes the server stuck on shutdown and the clients hung unexpected for about 30 seconds.
  • Hotrod clients block and receive a CacheException when stopping a cluster member of a JDG distributed cache
WARN  [Codec21] ISPN004005: Error received from the server: org.infinispan.commons.CacheException: Could not commit implicit transaction
  • The server which should be shut down hung for a few seconds and other server show messages like this
DEBUG [org.infinispan.server.hotrod.CacheDecodeContext] (HotRodServerWorker-2) Exception caught: org.infinispan.server.hotrod.HotRodException: java.security.PrivilegedActionException: org.infinispan.commons.CacheException: Could not commit implicit transaction
  at org.infinispan.server.hotrod.CacheDecodeContext.createServerException(CacheDecodeContext.scala:96) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
  at org.infinispan.server.hotrod.HotRodDecoder.decode(HotRodDecoder.scala:62) [infinispan-server-hotrod-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
  ...
Caused by: java.security.PrivilegedActionException: org.infinispan.commons.CacheException: Could not commit implicit transaction
  at org.infinispan.security.Security.doAs(Security.java:145) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
  ... 18 more
Caused by: org.infinispan.commons.CacheException: Could not commit implicit transaction
  at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1623) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
  at org.infinispan.CacheImpl.putIfAbsentInternal(CacheImpl.java:1105) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
  at org.infinispan.CacheImpl.putIfAbsent(CacheImpl.java:1094) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
  at org.infinispan.CacheImpl.putIfAbsent(CacheImpl.java:1678) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
  ... 20 more
Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
  at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1201)
  ... 31 more
Caused by: org.infinispan.commons.CacheException: Could not prepare.
  at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:47) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
  at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
)
  ... 34 more
Caused by: javax.transaction.xa.XAException
  at org.infinispan.transaction.TransactionCoordinator.rollback(TransactionCoordinator.java:184) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
  ... 39 more
Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 3
  at org.infinispan.statetransfer.StateTransferLockImpl.waitForTransactionData(StateTransferLockImpl.java:93) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
  ... 42 more

Environment

  • Red Hat JBoss Data Grid (JDG)
    • 6.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content