Replication timeout occurs during startup with server hinting
Issue
- Replication timeout occurs during startup of a node with server hinting setting.
- Starting up some RHDG nodes succeeded, but a specific number of nodes failed to start always with following error.
ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service jboss.datagrid-infinispan.clustered.<cache_name>: org.jboss.msc.service.StartException in service jboss.datagrid-infinispan.clustered.default: Failed to start service
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1978)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:878)
at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:643)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:630)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:232)
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:895)
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:678)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:622)
at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:508)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:494)
at org.infinispan.manager.impl.AbstractDelegatingEmbeddedCacheManager.getCache(AbstractDelegatingEmbeddedCacheManager.java:156)
at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:136)
at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:133)
at org.infinispan.security.Security.doPrivileged(Security.java:76)
at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:69)
at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:141)
at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:91)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:2032)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1955)
... 3 more
Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for svr01 (flags=0), site-id=site1, rack-id=rack1, machine-id=machine1)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:916)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$2(JGroupsTransport.java:654)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:51)
at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:16)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
... 3 more
- Sometimes the issue can be seen as Initial state transfer timeout.
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache <cache_name> on svr03 (flags=0), site-id=site1, rack-id=rack1, machine-id=machine1)
at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:234)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
... 23 more
Environment
- Red Hat JBoss Data Grid (RHDG)
- 7.x
- Red Hat JBoss Enterprise Application Platform (EAP)
- 7.2.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.