DG 8 cluster with memcached hangs at start phase

Solution Verified - Updated -

Issue

DG 8 with memcached deployed hangs on starting phase - thread dump shows waiting on parking eventually causes timeouts:

"non-blocking-thread--p2-t3" #27 daemon prio=5 os_prio=0 cpu=474.13ms elapsed=442.98s tid=0x00005646a0e15800 nid=0xbe waiting on condition  [0x00007fc1d5b3f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)
    - parking to wait for  <0x00000002c71d31e8> (a java.util.concurrent.CompletableFuture$Signaller) <--------------------------------------
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.16/LockSupport.java:234)
    at java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.16/CompletableFuture.java:1798)
    at java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.16/ForkJoinPool.java:3128)
    at java.util.concurrent.CompletableFuture.timedGet(java.base@11.0.16/CompletableFuture.java:1868)
    at java.util.concurrent.CompletableFuture.get(java.base@11.0.16/CompletableFuture.java:2021) <-------------------------------------------
    at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:126)
    at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:36)
    at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:249)
    at org.infinispan.cache.impl.InvocationHelper.doInvoke(InvocationHelper.java:297)
    at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:101)
    at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:83)
    at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1300)
    at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1828)
    at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.put(AbstractDelegatingAdvancedCache.java:325)
    at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:424)
    at org.infinispan.server.memcached.MemcachedDecoder.put(MemcachedDecoder.java:340)
    at org.infinispan.server.memcached.MemcachedDecoder.decodeValue(MemcachedDecoder.java:278)
    at org.infinispan.server.memcached.MemcachedDecoder.decodeDispatch(MemcachedDecoder.java:191)
    at org.infinispan.server.memcached.MemcachedDecoder.decode(MemcachedDecoder.java:135)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
    at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
    at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:28)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at java.lang.Thread.run(java.base@11.0.16/Thread.java:829)

Environment

  • Red hat OpenShift Container Platform (OCP)
    • 4.x
  • Red Hat Data Grid (RHDG)
    • 8.x
    • memcached

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content