Unaccounted memory usage by "zing_mm" module.
Environment
- Red Hat Enterprise Linux 6
- Custom module zing_mm.ko is loaded.
Issue
- System shows unaccounted memory
- The total used amount without cacehd is 717208, but checking the process usage shows only 125GB
$ cat sos_commands/memory/free_-m
total used free shared buffers cached
Mem: 775350 766598 8751 0 11 49377
-/+ buffers/cache: 717208 58141 <<<--------[ Actual Free memory.]
Swap: 4095 0 4095
$ cat ps | tail -n +2 | cut -b2- | gawk '{mem += $6} END {print "total " mem/1048576 "GB"}'
total 125.962GB
Resolution
- The unaccountable` memory on this system is due to zing_mm module.
- Unload "zing_mm" module and stop the "zing-memory" service.
$ grep -i zing chkconfig
zing-licensed 0:off 1:off 2:on 3:on 4:on 5:on 6:off
zing-memory 0:off 1:off 2:on 3:on 4:on 5:on 6:off
- Monitor the memory usage after unloading the module & stopping the service.
- Contact vendor of "zing_mm" module and take their opinion on this issue.
Root Cause
- The analysis of sosreport shows that Zing Memory Management Enhancements module (zing_mm) is loaded on this system.
$ grep zing_mm sos_commands/general/dmesg_now
zing_mm module is older than RHEL 6.2 ... applying fixups
zing_mm module is older than RHEL 6.2 ... applying fixups
$ head -n 8 etc/rc.d/init.d/zing-memory
#!/bin/bash
# the following is chkconfig init header
#
# zing-memory: pmem .ko module
#
# chkconfig: 2345 45 55
# description: Set up Zing memory kernel module
^
+----
$ grep zing_mm lsmod
zing_mm 386619 8
^
'....
$ grep zing_mm proc/kallsyms | head
ffffffffa02d1380 b replacement_binfmt [zing_mm]
ffffffffa02cfd10 r .LC0 [zing_mm]
ffffffffa02b8380 t zm_vma_dump_size [zing_mm]
ffffffffa02b83d0 t trace_kmalloc [zing_mm]
ffffffffa02b8460 t dump_write [zing_mm]
ffffffffa02b8520 t alignfile [zing_mm]
ffffffffa02cf400 r buf.35442 [zing_mm]
ffffffffa02b8590 t notesize [zing_mm]
ffffffffa02b85d0 t writenote [zing_mm]
ffffffffa02b86a0 t dump_seek [zing_mm]
- The module is stored inside "/lib/modules/zing-driver/extra" directory. Hence "modinfo" command failed to retrieve any information about this
module.
$ grep HACK etc/rc.d/init.d/zing-memory -A 5
# HACK!
KODIR=/lib/modules/zing-driver/extra
KOBASE=zing_mm
KO=$KOBASE.ko
KOFILE=$KODIR/$KO
$ grep -i zing sos_commands/kernel/modinfo_ip6_tables_ebtable_nat_ebtables_ipt_MASQUERADE_iptable_n
author:ERROR: modinfo: could not find module zing_mm
^
'....
$ cat etc/udev/rules.d/99-zing-md.rules
KERNEL=="zing_mm", MODE="0666", OWNER="root", GROUP="root"
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
