Long running RHDG 7.3 server with HotRod clients fail with OutOfMemoryError
Issue
- according to graphs, the heap memory has not stopped growing in all RHDG cluster instances until giving an OutOfMemoryError in one of them. The ERROR is different but the root cause is always
Caused by: java.lang.OutOfMemoryError: Java heap space
- HeapDump analysis after OutOfMemory shows the following classes are consuming the memory, the Thread shows
cacheStoppedListeners
like followed
Class Name | Shallow Heap | Retained Heap | Percentage
------------------------------------------------------------------------------------------------------------------------------------
org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl @ 0x760f67878| 48 | 820,481,576 | 96.28%
Class Name | Ref. Objects |
--------------------------------------------------------------------------------------------------------------------------------------------------
java.lang.Thread @ 0x760886888 timeout-thread--p3-t1 Native Stack, Thread | 1,841,206 |
'- <Java Local> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue @ 0x76088ab00 | 1,841,206 |
'- queue java.util.concurrent.RunnableScheduledFuture[24] @ 0x760ec5ef0 | 1,841,206 |
'- [0] java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask @ 0x7617bde40 | 1,841,206 |
'- callable java.util.concurrent.Executors$RunnableAdapter @ 0x7618b76c0 | 1,841,206 |
'- task org.infinispan.transaction.impl.TransactionTable$$Lambda$692 @ 0x7618b76d8 | 1,841,206 |
'- arg$1 org.infinispan.transaction.impl.TransactionTable @ 0x761617028 | 1,841,206 |
|- cacheManagerNotifier org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl @ 0x760f67878| 1,841,203 |
| |- cacheStoppedListeners java.util.concurrent.CopyOnWriteArrayList @ 0x762068998 | 1,841,183 |
| | '- array java.lang.Object[647063] @ 0x7951cb888 | 1,841,183 |
Environment
- Red Hat Data Grid (RHDG)
- 7.3
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.