A RHDG cluster seems blocked an not longer resposive if expiration with max-idle is used

Solution Verified - Updated -

Issue

  • A DG cluster will stuck and clients will not get any response, the grid is back to work if we kill one node which is suspected as root cause
  • Data Grid is completely unresponsive and there are blocked threads shown whithin the thread dumps like followed
"HotRod-hotrod-internalServerWorker-4-12" #279 prio=5 os_prio=0 tid=0x00007f35980b8800 nid=0xbbb waiting for monitor entry [0x00007f356cfb4000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8.compute(EquivalentConcurrentHashMapV8.java:1910)

    waiting to lock <0x0000000742cca028> (a org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8$Node)
    at org.infinispan.container.DefaultDataContainer.compute(DefaultDataContainer.java:335)
    at org.infinispan.expiration.impl.ExpirationManagerImpl.handleInMemoryExpiration(ExpirationManagerImpl.java:135)
    at org.infinispan.expiration.impl.ClusterExpirationManager.handleInMemoryExpiration(ClusterExpirationManager.java:152)
    locked <0x0000000742cc9898> (a org.infinispan.container.entries.metadata.MetadataTransientCacheEntry)
    at org.infinispan.container.DefaultDataContainer.get(DefaultDataContainer.java:201)

"pool-7-thread-1" #158 prio=5 os_prio=0 tid=0x00007f35c816c000 nid=0xb35 waiting for monitor entry [0x00007f3575238000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.infinispan.expiration.impl.ExpirationManagerImpl.lambda$handleInMemoryExpiration$0(ExpirationManagerImpl.java:137)

    waiting to lock <0x0000000742cc9898> (a org.infinispan.container.entries.metadata.MetadataTransientCacheEntry)
    at org.infinispan.expiration.impl.ExpirationManagerImpl$$Lambda$374/1904915245.compute(Unknown Source)
    at org.infinispan.container.DefaultDataContainer.lambda$compute$3(DefaultDataContainer.java:336)
    at org.infinispan.container.DefaultDataContainer$$Lambda$375/1917382785.apply(Unknown Source)
    at org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8.compute(EquivalentConcurrentHashMapV8.java:1919)
    locked <0x0000000742cca028> (a org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8$Node)
    at org.infinispan.container.DefaultDataContainer.compute(DefaultDataContainer.java:335)
    at org.infinispan.expiration.impl.ExpirationManagerImpl.handleInMemoryExpiration(ExpirationManagerImpl.java:135)
    at org.infinispan.expiration.impl.ClusterExpirationManager.processExpiration(ClusterExpirationManager.java:82)

Environment

  • Red Hat Data Grid (RHDG)
    • 7

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content