ISPN000136: Error executing command PutKeyValueCommand on Cache
Issue
- Multiple Nodes trying to update the same key at the same time. This can cause deadlocks, as each one locks the
key locally and then tries to replicate it. But the replication cannot acquire the lock on the remote node
that the other writer is holding. - DataGrid GUI is completely frozen when the locking issue occurs.
WARN org.infinispan.HOTROD.handleException - ISPN004098: Closing connection [id: XXXX, L:/y.y.y.y:61375 - R:/x.xx.x.x:11222] due to transport errorjava.net.SocketTimeoutException: PutAllOperation{user-cache, flags=6, connection=/x.x.x.x:11222} timed out after 60000 ms
at org.infinispan.client.hotrod.impl.operations.HotRodOperation.run(HotRodOperation.java:182)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
ERROR (non-blocking-thread-node-3-p2-t20) [org.infinispan.interceptors.impl.InvocationContextInterceptor] ISPN000136: Error executing command LockControlCommand on Cache 'XXX', writing keys [] org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 60 seconds for key WrappedByteArray[\J0F\2\3\3\4\1\1\4\0\7\4\4\6\8\9\2 (17 bytes)] and requestor GlobalTransaction{id=1690332, addr=node-1, remote=true, xid=null, internalId=-1}. Lock is held by GlobalTransaction{id=12345, addr=node-3, remote=false, xid=null, internalId=-1}
Environment
- Red Hat Data Grid
- 8.3.1
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.