Replication timeout occurs during startup with server hinting

Issue

• Replication timeout occurs during startup of a node with server hinting setting.
• Starting up some RHDG nodes succeeded, but a specific number of nodes failed to start always with following error.

ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service jboss.datagrid-infinispan.clustered.<cache_name>: org.jboss.msc.service.StartException in service jboss.datagrid-infinispan.clustered.default: Failed to start service
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1978)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
    at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172)
    at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:878)
    at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:643)
    at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:630)
    at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549)
    at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:232)
    at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:895)
    at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
    at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:678)
    at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:622)
    at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:508)
    at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:494)
    at org.infinispan.manager.impl.AbstractDelegatingEmbeddedCacheManager.getCache(AbstractDelegatingEmbeddedCacheManager.java:156)
    at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:136)
    at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:133)
    at org.infinispan.security.Security.doPrivileged(Security.java:76)
    at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:69)
    at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:141)
    at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:91)
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:2032)
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1955)
    ... 3 more
Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for svr01 (flags=0), site-id=site1, rack-id=rack1, machine-id=machine1)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:916)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$2(JGroupsTransport.java:654)
    at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
    at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:51)
    at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:16)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    ... 3 more

• Sometimes the issue can be seen as Initial state transfer timeout.

Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache <cache_name> on svr03 (flags=0), site-id=site1, rack-id=rack1, machine-id=machine1)
    at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:234)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
    ... 23 more