Concurrent singleton service installation can cause service to run simultaneously on 2 member in JBoss EAP 7 and Single Sign On 7

Issue

• Singleton service is deployed in all host in an EAP Domain a starts active in one of the host.
• Active Singleton host service is stopped and another host from Domain become active.
• Start again previous stopped host makes Singleton application active in 2 host members and is raised Replication timeout Exception:

ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ServerService Thread Pool -- 65) ISPN000136: Error executing command PrepareCommand, writing keys [host-name]: org.infinispan.util.concurrent.TimeoutException: Replication timeout
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:641)
        at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
        at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
        at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:61)
        at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: Replication timeout
                at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
...
       Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout
                at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:641)
                at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
...
        [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: Replication timeout]

• Replication error messages in the logs of two servers. After restarting both hosts, only one of them restarted propely. The other shows error messages and doesn't start.

2022-08-08 09:58:27,498 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 62) MSC000001: Failed to start service org.wildfly.clustering.cache.registry.ejb.client-mappings: org.jboss.msc.service.StartException in service org.wildfly.clustering.cache.registry.ejb.client-mappings: org.infinispan.commons.CacheException: javax.transaction.HeuristicMixedException
    at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:70)
    at org.wildfly.clustering.service.AsynchronousServiceBuilder.lambda$start$0(AsynchronousServiceBuilder.java:99)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: org.infinispan.commons.CacheException: javax.transaction.HeuristicMixedException
    at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:102)
    at org.wildfly.clustering.server.registry.CacheRegistry.populateRegistry(CacheRegistry.java:105)
    at org.wildfly.clustering.server.registry.CacheRegistry.<init>(CacheRegistry.java:98)
    at org.wildfly.clustering.server.registry.CacheRegistryFactoryBuilder.lambda$null$0(CacheRegistryFactoryBuilder.java:87)
    at org.wildfly.clustering.server.registry.FunctionalRegistryFactory.createRegistry(FunctionalRegistryFactory.java:50)
    at org.wildfly.clustering.server.registry.RegistryBuilder.lambda$build$0(RegistryBuilder.java:82)
    at org.wildfly.clustering.service.SuppliedValueService.lambda$new$0(SuppliedValueService.java:42)
    at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:68)
    ... 5 more
Caused by: javax.transaction.HeuristicMixedException
    at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:439)
    at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:448)
    at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:321)
    at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:108)
    at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
    ... 12 more
Caused by: javax.transaction.xa.XAException
    at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:207)
    at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)
    at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:114)
    at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:401)
    ... 16 more
Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:641)
    at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
    at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:61)
    at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: Replication timeout
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
        at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:100)
        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:187)
        at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:197)
        at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:187)
        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
        at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:108)
        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
        at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
        at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:37)
        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
        at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:93)
        at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:106)
        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
        at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:158)
        at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:145)
        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
        at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
        at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42)
        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
        at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:229)
        at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:87)
        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
        at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
        at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:79)
        at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
        at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
        at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)
        at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:157)
        at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:114)
        at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:401)
        at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:448)
        at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:321)
        at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:108)
        at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
        at org.wildfly.clustering.server.registry.CacheRegistry.populateRegistry(CacheRegistry.java:105)
        at org.wildfly.clustering.server.registry.CacheRegistry.<init>(CacheRegistry.java:98)
        at org.wildfly.clustering.server.registry.CacheRegistryFactoryBuilder.lambda$null$0(CacheRegistryFactoryBuilder.java:87)
        at org.wildfly.clustering.server.registry.FunctionalRegistryFactory.createRegistry(FunctionalRegistryFactory.java:50)
        at org.wildfly.clustering.server.registry.RegistryBuilder.lambda$build$0(RegistryBuilder.java:82)
        at org.wildfly.clustering.service.SuppliedValueService.lambda$new$0(SuppliedValueService.java:42)
        at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:68)
        at org.wildfly.clustering.service.AsynchronousServiceBuilder.lambda$start$0(AsynchronousServiceBuilder.java:99)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at org.jboss.threads.JBossThread.run(JBossThread.java:320)
    [CIRCULAR REFERENCE:org.infinispan.util.concurrent.TimeoutException: Replication timeout]

2022-08-08 09:58:27,548 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
    ("subsystem" => "infinispan"),
    ("cache-container" => "ejb")
]) - failure description: {"WFLYCTL0080: Failed services" => {"org.wildfly.clustering.cache.registry.ejb.client-mappings" => "org.infinispan.commons.CacheException: javax.transaction.HeuristicMixedException
    Caused by: org.infinispan.commons.CacheException: javax.transaction.HeuristicMixedException
    Caused by: javax.transaction.HeuristicMixedException
    Caused by: javax.transaction.xa.XAException
    Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout"}}
2022-08-08 09:58:27,592 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 27) WFLYSRV0010: Deployed "keycloak-server.war" (runtime-name : "keycloak-server.war")
2022-08-08 09:58:27,596 INFO  [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report
WFLYCTL0186:   Services which failed to start:      service org.wildfly.clustering.cache.registry.ejb.client-mappings: org.infinispan.commons.CacheException: javax.transaction.HeuristicMixedException