"ISPN000136: Error executing command PrepareCommand..." and "ISPN000482: Cannot create remote transaction GlobalTx:xxx:xxx, already completed" happened in JBoss EAP 7.x

Issue

• The first node gets the ERROR message "ISPN000136: Error executing command PrepareCommand, writing keys..." as follows.

  ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (thread-12,ejb,node1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(V1ID7_aYhdl5wEYbq6li-737JVXm37M7M8SXSdtG), SessionAttributesKey(V1ID7_aYhdl5wEYbq6li-737JVXm37M7M8SXSdtG), SessionAccessMetaDataKey(V1ID7_aYhdl5wEYbq6li-737JVXm37M7M8SXSdtG)]: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node2, see cause for remote stack trace
      at org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27)
      at org.infinispan.remoting.transport.impl.MapResponseCollector.addException(MapResponseCollector.java:65)
      at org.infinispan.remoting.transport.impl.MapResponseCollector$IgnoreLeavers.addException(MapResponseCollector.java:103)
      at org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29)
      at org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91)
      at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1370)
      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1273)
      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:125)
      at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1418)
      at org.jgroups.JChannel.up(JChannel.java:816)
      at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:133)
      at org.jgroups.stack.Protocol.up(Protocol.java:340)
      at org.jgroups.protocols.FORK.up(FORK.java:141)
      at org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
      at org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
      at org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
      at org.jgroups.protocols.pbcast.GMS.up(GMS.java:873)
      at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
      at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1003)
      at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:729)
      at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:384)
      at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
      at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:130)
      at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:203)
      at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:253)
      at org.jgroups.protocols.MERGE3.up(MERGE3.java:280)
      at org.jgroups.protocols.Discovery.up(Discovery.java:269)
      at org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
      at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
      at java.lang.Thread.run(Thread.java:748)
  Caused by: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:node1:13, already completed
      at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:388)
      at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
      at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:381)
      at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:361)
      at org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:121)
      at org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:102)
      at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
      at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
      at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
      at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
      ... 1 more
  

• In other nodes might get another WARN message "ISPN000482: Cannot create remote transaction GlobalTx:xxx:xx, already completed..." as follows.

  WARN  [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p7-t3) ISPN000071: Caught exception when handling command PrepareCommand {modifications=[PutKeyValueCommand{key=SessionCreationMetaDataKey(V1ID7_aYhdl5wEYbq6li-737JVXm37M7M8SXSdtG), value=org.wildfly.clustering.web.infinispan.session.SessionCreationMetaDataEntry@44abf95, flags=[FORCE_SYNCHRONOUS], commandInvocationId=CommandInvocation:local:0, putIfAbsent=true, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=5}, PutKeyValueCommand{key=SessionAccessMetaDataKey(V1ID7_aYhdl5wEYbq6li-737JVXm37M7M8SXSdtG), value=org.wildfly.clustering.web.infinispan.session.SimpleSessionAccessMetaData@4e718207, flags=[IGNORE_RETURN_VALUES], commandInvocationId=CommandInvocation:local:0, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=5}, PutKeyValueCommand{key=SessionAttributesKey(V1ID7_aYhdl5wEYbq6li-737JVXm37M7M8SXSdtG), value=[B@70412632, flags=[IGNORE_RETURN_VALUES], commandInvocationId=CommandInvocation:local:0, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=5}], onePhaseCommit=true, retried=false, gtx=GlobalTx:node1:13, cacheName='sample.war', topologyId=5}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:node1:13, already completed
      at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:388)
      at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
      at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:381)
      at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:361)
      at org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:121)
      at org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:102)
      at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
      at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
      at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
      at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
      at java.lang.Thread.run(Thread.java:748)