systemd[xxx]: fatal error: out of memory allocating heap arena map

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 8

Issue

  • Memory runtime errors such as out of memory & Cannot allocate memory even when there is no memory high consumption and no saturation of pid_max

Resolution

  • There was a limit set on data ulimit and replacing it with a higher or unlimited value resolves the issue.
# cat etc/security/limits.d/10-limit.conf 
<application>     soft    data        unlimited
<application>     hard    data        unlimited
<user>            soft    data        unlimited
<user>            hard    data        unlimited

Root Cause

  • Due to the limit set on data, the runtime memory was giving Out of Memory

Diagnostic Steps

  • Following messages observed in var/log/messages
$ grep -i "out of memory" messages
 systemd[224072]: fatal error: out of memory allocating heap arena map
 systemd[236702]: fatal error: out of memory allocating heap arena map
 systemd[251112]: fatal error: out of memory allocating heap arena map
 systemd[254668]: fatal error: out of memory allocating heap arena map
 systemd[255489]: fatal error: out of memory allocating heap arena map
 systemd[255891]: fatal error: out of memory allocating heap arena map
 journal[256114]: out of memory [256114]
 systemd[256408]: fatal error: out of memory allocating heap arena map
 systemd[259592]: fatal error: out of memory allocating heap arena map
 systemd[259731]: fatal error: out of memory allocating heap arena map
 systemd[262546]: fatal error: out of memory allocating heap arena map
 systemd[264903]: fatal error: out of memory allocating heap arena map
 journal[267174]: out of memory [267174]
 journal[267200]: out of memory [267200]
 systemd[269696]: fatal error: out of memory allocating heap arena map
 systemd[271157]: fatal error: out of memory allocating heap arena map
 systemd[271508]: fatal error: out of memory allocating heap arena map
 systemd[271758]: fatal error: out of memory allocating heap arena map
 systemd[271851]: fatal error: out of memory allocating heap arena map
 systemd[271889]: fatal error: out of memory allocating heap arena map
  • Analyzing memory and plist-sz from sar data, shows resource utilization is normal

Memory

◯ memory utilization is normal

12:00:02 Am kbmemfree   kbavail kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty 
Average:      2128814   2849500  29176931     89.35     10288   1111895   2143985      5.82    637520    819366         1

Load Average

◯ plist-sz is within limits

12:00:02 Am   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
Average:            8       514      0.00      0.00      0.00         0
  • Check For Application Ulimit-data:
# cat etc/security/limits.d/<application>.conf 
<application>     soft    data        2097152
<application>     hard    data        2097152
<application>     soft    data        204800
<application>     hard    data        204800

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments