We are suspecting memory leak in our critical production system

Solution Verified - Updated -

Issue

  • It seems one of our application server going high memory utilization (above 80%) suspecting there is memory leak.
  • We noticed total 32 gb of memory, 11gb allocated for apps and 7gb is free, need to know about 14gb of balance utilization.
  • Please note that the unaccounted memory size varies throughout the day. As of now we see approximately 7.93 GiB memory is unaccounted, approximately 13 GB is used by processes, and 2 GB is used by unreclaimable slab.
$ cat sos_commands/memory/free_-m 

              total        used        free      shared  buff/cache   available
Mem:          31884       23889        2542         412        5453        7071   ----------->> Available Memory
Swap:          4095        2234        1861
  • Total used RAM on system is '23.79' GiB and the memory consumed by the process is '13.8' GiB:
$ cat ps | gawk '{sum+=$6} END {print "Total RSS : " sum/1024^2 " GiB"}'
Total RSS : 13.8036 GiB
  • The top MEM-eating threads are:
    USER      PID    %CPU  %MEM  VSZ-MiB  RSS-MiB  TTY    STAT   START  TIME     COMMAND  
    iloadadm  29637  18.5  16.6  15791    5297     ?      -      Jun22  8601:15  /usr/java/latest/bin/java -DEPOD_MODULE_NAME=ILOAD2 -server 
    iloadadm  24463  9.2   14.6  14565    4678     ?      -      Jun22  4301:18  /usr/java/latest/bin/java -DEPOD_MODULE_NAME=ILOAD2A -server 
    iloadadm  8841   3.2   2.8   10979    919      ?      -      Jul19  227:15   /usr/java/latest/bin/java -DEPOD_MODULE_NAME=ILOAD2SFTP -server 
    iloadadm  15778  0.2   1.2   5110     402      ?      -      02:38  0:32     /usr/java/latest/bin/java -DEPOD_MODULE_NAME=ILOAD2RECO -server 
    root      1772   24.6  1.2   4870     401      ?      -      04:23  39:23    /opt/traps/bin/pmd 
    root      14687  0.0   0.8   1309     277      ?      -      Jun22  0:00     /sbin/vxencryptd -m 
    root      10303  40.3  0.7   473      249      pts/3  -      07:03  0:03     /usr/bin/python /usr/lib/python2.7/site-packages/insights_client/run.py --offline 
    root      5631   44.9  0.4   1168     144      pts/3  -      07:03  0:22     /usr/bin/python /usr/sbin/sosreport 
    root      23820  0.0   0.4   225      133      ?      -      Jun22  29:19    /opt/VRTSvcs/bin/had 
    root      9649   0.7   0.4   1534     129      ?      -      Jun22  364:19   /opt/ds_agent/ds_agent -w /var/opt/ds_agent 
  • The slab is around '2.65' GiB:
$ cat proc/meminfo 
Slab:            2659304 kB    --------->>
SReclaimable:     492752 kB
SUnreclaim:      2166552 kB
  • So we suspect there is something running on our system which we are not able to catch through the usual means.

Environment

  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 9
  • Red Hat Enterprise Linux 10

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content