We are suspecting memory leak in our critical production system
Issue
- It seems one of our application server going high memory utilization (above 80%) suspecting there is memory leak.
- We noticed total 32 gb of memory, 11gb allocated for apps and 7gb is free, need to know about 14gb of balance utilization.
- Please note that the unaccounted memory size varies throughout the day. As of now we see approximately 7.93 GiB memory is unaccounted, approximately 13 GB is used by processes, and 2 GB is used by unreclaimable slab.
$ cat sos_commands/memory/free_-m
total used free shared buff/cache available
Mem: 31884 23889 2542 412 5453 7071 ----------->> Available Memory
Swap: 4095 2234 1861
- Total used RAM on system is '23.79' GiB and the memory consumed by the process is '13.8' GiB:
$ cat ps | gawk '{sum+=$6} END {print "Total RSS : " sum/1024^2 " GiB"}'
Total RSS : 13.8036 GiB
- The top MEM-eating threads are:
USER PID %CPU %MEM VSZ-MiB RSS-MiB TTY STAT START TIME COMMAND
iloadadm 29637 18.5 16.6 15791 5297 ? - Jun22 8601:15 /usr/java/latest/bin/java -DEPOD_MODULE_NAME=ILOAD2 -server
iloadadm 24463 9.2 14.6 14565 4678 ? - Jun22 4301:18 /usr/java/latest/bin/java -DEPOD_MODULE_NAME=ILOAD2A -server
iloadadm 8841 3.2 2.8 10979 919 ? - Jul19 227:15 /usr/java/latest/bin/java -DEPOD_MODULE_NAME=ILOAD2SFTP -server
iloadadm 15778 0.2 1.2 5110 402 ? - 02:38 0:32 /usr/java/latest/bin/java -DEPOD_MODULE_NAME=ILOAD2RECO -server
root 1772 24.6 1.2 4870 401 ? - 04:23 39:23 /opt/traps/bin/pmd
root 14687 0.0 0.8 1309 277 ? - Jun22 0:00 /sbin/vxencryptd -m
root 10303 40.3 0.7 473 249 pts/3 - 07:03 0:03 /usr/bin/python /usr/lib/python2.7/site-packages/insights_client/run.py --offline
root 5631 44.9 0.4 1168 144 pts/3 - 07:03 0:22 /usr/bin/python /usr/sbin/sosreport
root 23820 0.0 0.4 225 133 ? - Jun22 29:19 /opt/VRTSvcs/bin/had
root 9649 0.7 0.4 1534 129 ? - Jun22 364:19 /opt/ds_agent/ds_agent -w /var/opt/ds_agent
- The slab is around '2.65' GiB:
$ cat proc/meminfo
Slab: 2659304 kB --------->>
SReclaimable: 492752 kB
SUnreclaim: 2166552 kB
- So we suspect there is something running on our system which we are not able to catch through the usual means.
Environment
- Red Hat Enterprise Linux 7
- Red Hat Enterprise Linux 8
- Red Hat Enterprise Linux 9
- Red Hat Enterprise Linux 10
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.