Long running RHDG 7.3 server with HotRod clients fail with OutOfMemoryError

Solution Verified - Updated -

Issue

  • according to graphs, the heap memory has not stopped growing in all RHDG cluster instances until giving an OutOfMemoryError in one of them. The ERROR is different but the root cause is always
Caused by: java.lang.OutOfMemoryError: Java heap space
  • HeapDump analysis after OutOfMemory shows the following classes are consuming the memory, the Thread shows cacheStoppedListeners like followed
Class Name                                                                              | Shallow Heap | Retained Heap | Percentage
------------------------------------------------------------------------------------------------------------------------------------
org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl @ 0x760f67878|           48 |   820,481,576 |     96.28%
Class Name                                                                                                                        | Ref. Objects | 
--------------------------------------------------------------------------------------------------------------------------------------------------
java.lang.Thread @ 0x760886888  timeout-thread--p3-t1 Native Stack, Thread                                                        |    1,841,206 |
'- <Java Local> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue @ 0x76088ab00                                   |    1,841,206 |
   '- queue java.util.concurrent.RunnableScheduledFuture[24] @ 0x760ec5ef0                                                        |    1,841,206 |
      '- [0] java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask @ 0x7617bde40                                   |    1,841,206 |
         '- callable java.util.concurrent.Executors$RunnableAdapter @ 0x7618b76c0                                                 |    1,841,206 |
            '- task org.infinispan.transaction.impl.TransactionTable$$Lambda$692 @ 0x7618b76d8                                    |    1,841,206 |
               '- arg$1 org.infinispan.transaction.impl.TransactionTable @ 0x761617028                                            |    1,841,206 |
                  |- cacheManagerNotifier org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl @ 0x760f67878|    1,841,203 |
                  |  |- cacheStoppedListeners java.util.concurrent.CopyOnWriteArrayList @ 0x762068998                             |    1,841,183 |
                  |  |  '- array java.lang.Object[647063] @ 0x7951cb888                                                           |    1,841,183 |

Environment

  • Red Hat Data Grid (RHDG)
    • 7.3

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content