Unaccountable memory usage in RHEL by zing memory module [zing_mm]

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 8
  • Azul Zulu Prime System Tools: zing-zst
  • Azul Zulu Prime System Zing memory module: [zing_mm]

Issue

  • Unaccountable high memory usage when nothing is running on the server. The total used memory without Buffer, Cache, and Slab is 60 GiB, but checking the process memory usage shows only 1 GiB.
# free 
              total        used        free      shared  buff/cache   available
Mem:       73958308    62946516      785844         784    10225948    10219380
Swap:      16777212           0    16777212

Total memory usage from user-space is 1.0 GiB.

$ cat ps | tail -n +2 | cut -b2- | gawk '{mem += $6} END {print "Total RSS of user-mode: " mem/1048576 " GiB"}'
Total RSS of user-mode: 1.01976 GiB

The memory used for Buffer/Cache is 8.87 GiB, and the memory used for Slab is 1.16 GiB.

$ cat /proc/meminfo 
MemTotal:       73958308 kB
MemFree:          785700 kB
MemAvailable:   10218132 kB
Buffers:           28996 kB
Cached:          9273168 kB
SwapCached:            0 kB
Active:           890280 kB
Inactive:        8830200 kB
Active(anon):     418980 kB
Inactive(anon):      120 kB
Active(file):     471300 kB
Inactive(file):  8830080 kB
Unevictable:          32 kB
Mlocked:              32 kB
SwapTotal:      16777212 kB
SwapFree:       16777212 kB
Dirty:             58364 kB
Writeback:             0 kB
AnonPages:        410104 kB
Mapped:           161584 kB
Shmem:               784 kB
KReclaimable:     922676 kB
Slab:            1214228 kB
SReclaimable:     922676 kB
SUnreclaim:       291552 kB
KernelStack:       12820 kB
PageTables:        17568 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    53756364 kB
Committed_AS:    3849196 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:             5440 kB
HardwareCorrupted:     0 kB
AnonHugePages:    147456 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      319360 kB
DirectMap2M:     9117696 kB
DirectMap1G:    68157440 kB

Resolution

  • Contact Azul support and take their opinion on unaccountable memory reserved by the System Zing Memory module [zing_mm].

Additional Information:

  • You can obtain detailed memory usage information using the following command.
# zing-ps -s

Refer to the Azul Zulu Prime System document: zing-ps Command Output

  • Stopping the System Zing Memory service (zing-memory) will release the reserved memory.
# service zing-memory stop
# systemctl stop zing-memory

Refer to the Azul Zulu Prime System document: Starting and Stopping Zing Memory

Root Cause

  • Unaccountable memory is reserved by Azul Zulu Prime Zing memory module [zing_mm] for the Java heap.
From: /var/log/messages

zing-memory[1227]: INFO: az_pmem_fund_transfer (to 0, from 7, bytes 60129542144) succeeded     
zing-configure-pmem[1733]: LOG_INFO: You can now run Java processes up to -Xmx57344m or -Xmx56g

$ bc -q
scale=2
60129542144/2^30
56.00 (GiB)

Refer to the Azul Zulu Prime System document: System Zing Memory

Diagnostic Steps

Red Hat Enterprise Linux 7 and 8:

  • Check if the Zing Memory module [zing_mm] is loaded.
$ grep zing_mm proc/modules 
zing_mm 8417280 0 - Live 0xffffffffc0834000 (OE)
  • Check if the "zing-memory" service is enabled and loaded.
$ egrep "zing-memory|UNIT" sos_commands/systemd/systemctl_list-units
  UNIT                           LOAD   ACTIVE     SUB    DESCRIPTION                                                                  
  zing-memory.service          loaded   active  exited    zing-memory service script      

$ cat usr/lib/systemd/system/zing-memory.service
[Unit]
Description=zing-memory service script

[Service]
Type=oneshot
ExecStart=/etc/init.d/zing-memory start
ExecStop=/etc/init.d/zing-memory stop
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
  • Check the "zing-memory" service information in the "/var/log/messages" to determine the memory reserved by the Zing Memory module [zing_mm].
kernel: zing_mm: loading out-of-tree module taints kernel.
kernel: zing_mm: module verification failed: signature and/or required key missing - tainting kernel
kernel: Zing memory management driver build zst-5.22.9.0-1, api version 8c
...
zing-memory[1227]: INFO:  Zing Memory reserved using reserve-at-config policy.
zing-memory[1227]: INFO:  azulPmemPages: 30208.
zing-configure-pmem[1733]: LOG_INFO: Zing Memory reserved using reserve-at-config policy
zing-configure-pmem[1733]: LOG_INFO: azulPmemPages: 30208
/usr/sbin/zing-licensed[1745]: License Daemon Initialized : 5.22.9.0-1
/usr/sbin/zing-licensed[1745]: Error: Unable to  change directory: No such file or directory
zing-licensem[1746]: Error: Unable to  change directory: No such file or directory
zing-licensed[1229]: [  OK  ]#015zing-licensed: INFO: start successful
systemd[1]: Started zing-licensed service script.
zing-memory[1227]: INFO: az_pmem_reserve_pages (num2mPages 30208) succeeded
zing-llpmem[1740]: INFO: az_pmem_reserve_pages (num2mPages 30208) succeeded
kernel: AZMM: Requested 30208 pages, allocated 30208 pages
kernel: AZMM: Timeout 100000 msec, elapsed 571 msec
kernel: AZMM: First attempt allocs from current node 30208
kernel: AZMM: First attempt allocs from all nodes 0
kernel: AZMM: Allocs from current node after compaction 0
kernel: AZMM: Allocs from all nodes after compaction 0
kernel: AZMM: Allocs from current node after dropping caches 0
kernel: AZMM: Allocs from current node after second compaction 0
kernel: AZMM: Final attempt allocs from all nodes 0
zing-llpmem[1754]: INFO: az_pmem_fund_transfer (to 7, from 0, bytes 63350767616) succeeded
zing-memory[1227]: INFO: az_pmem_fund_transfer (to 7, from 0, bytes 63350767616) succeeded
zing-llpmem[1755]: INFO: az_pmem_fund_transfer (to 1, from 7, bytes 1073741824) succeeded
zing-memory[1227]: INFO: az_pmem_fund_transfer (to 1, from 7, bytes 1073741824) succeeded
zing-llpmem[1757]: INFO: az_pmem_fund_transfer (to 3, from 7, bytes 2147483648) succeeded
zing-memory[1227]: INFO: az_pmem_fund_transfer (to 3, from 7, bytes 2147483648) succeeded
zing-llpmem[1759]: INFO: az_pmem_fund_transfer (to 0, from 7, bytes 60129542144) succeeded
zing-memory[1227]: INFO: az_pmem_fund_transfer (to 0, from 7, bytes 60129542144) succeeded       <<< - - - 
zing-configure-pmem[1733]: LOG_INFO: You can now run Java processes up to -Xmx57344m or -Xmx56g  <<< - - -
zing-memory[1227]: INFO:  You can now run Java processes up to -Xmx57344m or -Xmx56g.
zing-memory[1227]: INFO:  Azul pmem initialized successfully.
zing-configure-pmem[1733]: LOG_INFO: Azul pmem initialized successfully
zing-memory[1762]: INFO: start successful
zing-memory[1227]: zing-memory: INFO: start successful
systemd[1]: Started zing-memory service script.                                                   <<< - - -
  • The memory reserved by the Zing Memory module [zing_mm] module is 56 GiB.
zing-memory[1227]: INFO: az_pmem_fund_transfer (to 0, from 7, bytes 60129542144) succeeded     
                                                                    ^^^^^^^^^^^  
zing-configure-pmem[1733]: LOG_INFO: You can now run Java processes up to -Xmx57344m or -Xmx56g

$ bc -q
scale=2
60129542144/2^30
56.00 (GiB)
  • Check the "zing-zst" package information.
$ grep zing sos_commands/rpm/package-data  | awk '{print $1 "  " $8 " " $9}' 
zing-zst-8c.4.18.0-5.22.9.0.1.5.el8.x86_64  (none) ip-172-31-60-159.us-west-2.compute.internal

Red Hat Enterprise Linux 6:

  • Check if the Zing Memory module [zing_mm] is loaded.
# grep zing_mm /proc/modules 
zing_mm 10004873 0 - Live 0xffffffffa1449000
  • Check the Zing Memory module [zing_mm] information.
$ grep zing_mm sos_commands/kernel/modinfo* -A 13
filename:       /lib/modules/2.6.32-754.48.1.el6.x86_64/extra/zing_mm.ko
license:        GPL
description:    Zing memory management enhancements
author:         Gil Tene, Venkatram Tummala
zmd_version:    zst-5.21.8.0-1
srcversion:     4006FB52896F0E9368B78B9
depends:        
vermagic:       2.6.32-131.0.15.el6.x86_64 SMP mod_unload modversions 
parm:           zm_param_init_mm:ulong
parm:           zm_param_max_pfn:ulong
parm:           zm_param_phys_base:ulong
parm:           zm_param_sys_call_table:ulong
parm:           zm_param_vm_area_cachep:ulong
parm:           zm_param_current_kprobe:ulong
  • Check if the "zing-memory" service is enabled.
$ grep zing-memory sos_commands/startup/chkconfig_--list
zing-memory     0:off   1:off   2:on    3:on    4:on    5:on    6:off
  • Check the "zing-memory" service information in the "/var/log/messages" to determine the memory reserved by the Zing Memory module [zing_mm].
zing-memory: INFO: Starting...
kernel: Spectre V2 : WARNING: module 'zing_mm' built without retpoline-enabled compiler, may affect Spectre v2 mitigation
kernel: zing_mm: applying 16k kernel stack fix up
kernel: zing_mm: applying 16k kernel stack fix up
kernel: zing_mm: applying 16k kernel stack fix up
kernel: zing_mm: applying 16k kernel stack fix up
kernel: zing_mm: applying 16k kernel stack fix up
kernel: zing_mm: applying 16k kernel stack fix up
kernel: zing_mm: applying 16k kernel stack fix up
kernel: Zing memory management driver build zst-5.21.8.0-1, api version 8a
kernel: AZMM: Inited section 0 root[0]: ffff884023661000
kernel: AZMM: Inited section 512 root[1]: ffff88400a4ca000
kernel: AZMM: Inited section 1024 root[2]: ffff883ff166d000
kernel: AZMM: Inited section 1536 root[3]: ffff8840112e3000
kernel: AZMM: Inited section 2048 root[4]: ffff883ff1570000
kernel: AZMM: Inited section 2560 root[5]: ffff883ff5120000
kernel: AZMM: Inited section 3072 root[6]: ffff883ff5b23000
kernel: AZMM: Inited section 3584 root[7]: ffff883ff5b24000
kernel: AZMM: Inited section 4096 root[8]: ffff883ff5b1d000
kernel: AZMM: Inited 4096 vmemmap sections
zing-configure-pmem[9588]: LOG_INFO: Zing Memory reserved using reserve-at-config policy
zing-configure-pmem[9588]: LOG_INFO: azulPmemPages: 204800
kernel: AZMM: Total Requested 204800 pages, Allocated 204800 pages
kernel: AZMM: No Escalation allocs from current node 204800
kernel: AZMM: No Escalation allocs from all nodes 0
kernel: AZMM: allocs after compacting memory 0
kernel: AZMM: allocs after drop caches 0
kernel: AZMM: allocs after drop caches & compacting 0
kernel: AZMM: allocs from all nodes after all escalations 0
zing-llpmem: INFO: az_pmem_reserve_pages (num2mPages 204800) succeeded
zing-llpmem: INFO: az_pmem_fund_transfer (to 7, from 0, bytes 429496729600) succeeded
zing-llpmem: INFO: az_pmem_fund_transfer (to 1, from 7, bytes 42949672960) succeeded
zing-llpmem: INFO: az_pmem_fund_transfer (to 3, from 7, bytes 21474836480) succeeded
zing-llpmem: INFO: az_pmem_fund_transfer (to 0, from 7, bytes 365072220160) succeeded  <<< - - -
zing-configure-pmem[9588]: LOG_INFO: You can now run Java processes up to -Xmx348160m or -Xmx340g
zing-configure-pmem[9588]: LOG_INFO: Azul pmem initialized successfully
zing-memory: INFO: start successful
  • The memory reserved by the Zing Memory module [zing_mm] module is 340 GiB.
zing-llpmem: INFO: az_pmem_fund_transfer (to 0, from 7, bytes 365072220160) succeeded  <<< - - -
zing-configure-pmem[9588]: LOG_INFO: You can now run Java processes up to -Xmx348160m or -Xmx340g

$ bc -q
scale=2
365072220160/2^30
340.00 (GiB)
  • Check the "zing-zst" package information.
$ grep zing-zst sos_commands/rpm/package-data  | awk '{print $1 "  " $8 " " $9}' 
zing-zst-8a.2.6.32-5.21.8.0.1.el6.x86_64  (none) chicken.azulsystems.com

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments