Translated message

A translation of this page exists in English.

DistributedExecutorService が putAll を実行すると TimeoutException が発生する

Solution In Progress - Updated -

Environment

  • Red Hat JBoss Data Grid (JDG)
    • 6.x

Issue

  • 各ノードの cache.putAll(...) で、以下のスタックトレースが発生します。
2016-02-16 13:29:13,662 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread-52,node01) ISPN000136:Error executing command PutMapCommand, writing keys [AAAA, BBBB, CCCC, DDDD]: org.infinispan.remoting.RemoteException:Exception while processing put on primary owner
    at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutMapCommand(NonTxDistributionInterceptor.java:119) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:55) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:366) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForPutMapCommand(EntryWrappingInterceptor.java:437) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutMapCommand(EntryWrappingInterceptor.java:285) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:55) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.xsite.NonTransactionalBackupInterceptor.handleMultipleKeysWriteCommand(NonTransactionalBackupInterceptor.java:72) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.xsite.NonTransactionalBackupInterceptor.visitPutMapCommand(NonTransactionalBackupInterceptor.java:55) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:55) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutMapCommand(NonTransactionalLockingInterceptor.java:77) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:55) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:340) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:272) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.statetransfer.StateTransferInterceptor.visitPutMapCommand(StateTransferInterceptor.java:128) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:55) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutMapCommand(CacheMgmtInterceptor.java:151) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:55) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:105) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:74) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:63) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:55) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1610) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.CacheImpl.putAllInternal(CacheImpl.java:1130) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.CacheImpl.putAll(CacheImpl.java:1118) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.DecoratedCache.putAll(DecoratedCache.java:431) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at com.example.MyCallable.call(MyCallable.java:100) [classes:]
    at org.infinispan.commands.read.DistributedExecuteCommand.perform(DistributedExecuteCommand.java:98) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:92) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:47) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:185) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_60]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_60]
    at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_60]
Caused by: org.infinispan.remoting.RemoteException:ISPN000217:Received exception from node02-25334, see cause for remote stack trace
    at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:46) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:410) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:210) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:572) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:292) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.remoting.rpc.RpcManagerImpl$3.call(RpcManagerImpl.java:410) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_60]
    ...3 more
Caused by: org.infinispan.util.concurrent.TimeoutException:Unable to acquire lock after [20 seconds] on key [CCCC] for requestor [Thread[remote-thread-211,node02,5,ServerService ThreadGroup]]!Lock held by [Thread[remote-thread-79,node02,5,ServerService ThreadGroup]]
    at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:198) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLockNoCheck(LockManagerImpl.java:181) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:128) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutMapCommand(NonTransactionalLockingInterceptor.java:74) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:55) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:334) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:272) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.statetransfer.StateTransferInterceptor.visitPutMapCommand(StateTransferInterceptor.java:128) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:55) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutMapCommand(CacheMgmtInterceptor.java:151) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:55) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:105) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:74) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:63) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:55) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:92) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:47) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:185) [infinispan-core-6.4.0.Final-redhat-4.jar:6.4.0.Final-redhat-4]
    ...3 more

Resolution

  • キーセットが重複していないことを確認します。
  • キーセットの一意性を保証するために、org.infinispan.distexec.DistributedCallable#setEnvironment(Cache<K,V> cache, Set<K> inputKeys) 1 の inputKeys を使用することを検討してください。

  1. API document of org.infinispan.distexec.DistributedCallable ↩︎

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments