DG 8 cluster with memcached hangs at start phase
Issue
DG 8 with memcached deployed hangs on starting phase - thread dump shows waiting on parking eventually causes timeouts:
"non-blocking-thread--p2-t3" #27 daemon prio=5 os_prio=0 cpu=474.13ms elapsed=442.98s tid=0x00005646a0e15800 nid=0xbe waiting on condition [0x00007fc1d5b3f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)
- parking to wait for <0x00000002c71d31e8> (a java.util.concurrent.CompletableFuture$Signaller) <--------------------------------------
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.16/LockSupport.java:234)
at java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.16/CompletableFuture.java:1798)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.16/ForkJoinPool.java:3128)
at java.util.concurrent.CompletableFuture.timedGet(java.base@11.0.16/CompletableFuture.java:1868)
at java.util.concurrent.CompletableFuture.get(java.base@11.0.16/CompletableFuture.java:2021) <-------------------------------------------
at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:126)
at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:36)
at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:249)
at org.infinispan.cache.impl.InvocationHelper.doInvoke(InvocationHelper.java:297)
at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:101)
at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:83)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1300)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1828)
at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.put(AbstractDelegatingAdvancedCache.java:325)
at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:424)
at org.infinispan.server.memcached.MemcachedDecoder.put(MemcachedDecoder.java:340)
at org.infinispan.server.memcached.MemcachedDecoder.decodeValue(MemcachedDecoder.java:278)
at org.infinispan.server.memcached.MemcachedDecoder.decodeDispatch(MemcachedDecoder.java:191)
at org.infinispan.server.memcached.MemcachedDecoder.decode(MemcachedDecoder.java:135)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:28)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)
Environment
- Red hat OpenShift Container Platform (OCP)
- 4.x
- Red Hat Data Grid (RHDG)
- 8.x
- memcached
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.