Unaccounted memory usage by "zing_mm" module.

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 6
  • Custom module zing_mm.ko is loaded.

Issue

  • System shows unaccounted memory
  • The total used amount without cacehd is 717208, but checking the process usage shows only 125GB
$ cat sos_commands/memory/free_-m 
             total       used       free     shared    buffers     cached
Mem:        775350     766598       8751          0         11      49377
-/+ buffers/cache:     717208      58141  <<<--------[ Actual Free memory.]
Swap:         4095          0       4095

$ cat ps | tail -n +2 | cut -b2- | gawk '{mem += $6} END {print "total " mem/1048576 "GB"}'
total 125.962GB

Resolution

  • The unaccountable` memory on this system is due to zing_mm module.
  • Unload "zing_mm" module and stop the "zing-memory" service.
$ grep -i zing chkconfig 
zing-licensed   0:off   1:off   2:on    3:on    4:on    5:on    6:off
zing-memory     0:off   1:off   2:on    3:on    4:on    5:on    6:off
  • Monitor the memory usage after unloading the module & stopping the service.
  • Contact vendor of "zing_mm" module and take their opinion on this issue.

Root Cause

  • The analysis of sosreport shows that Zing Memory Management Enhancements module (zing_mm) is loaded on this system.
$ grep zing_mm sos_commands/general/dmesg_now 
zing_mm module is older than RHEL 6.2 ... applying fixups
zing_mm module is older than RHEL 6.2 ... applying fixups

$ head -n 8 etc/rc.d/init.d/zing-memory
#!/bin/bash

# the following is chkconfig init header
#
# zing-memory:   pmem .ko module
#
# chkconfig: 2345 45 55
# description:  Set up Zing memory kernel module         
                              ^
                              +----

$ grep zing_mm lsmod 
zing_mm               386619  8 
   ^
   '....

$ grep zing_mm proc/kallsyms | head
ffffffffa02d1380 b replacement_binfmt   [zing_mm]
ffffffffa02cfd10 r .LC0 [zing_mm]
ffffffffa02b8380 t zm_vma_dump_size [zing_mm]
ffffffffa02b83d0 t trace_kmalloc    [zing_mm]
ffffffffa02b8460 t dump_write   [zing_mm]
ffffffffa02b8520 t alignfile    [zing_mm]
ffffffffa02cf400 r buf.35442    [zing_mm]
ffffffffa02b8590 t notesize [zing_mm]
ffffffffa02b85d0 t writenote    [zing_mm]
ffffffffa02b86a0 t dump_seek    [zing_mm]
  • The module is stored inside "/lib/modules/zing-driver/extra" directory. Hence "modinfo" command failed to retrieve any information about this
    module.
$ grep HACK etc/rc.d/init.d/zing-memory -A 5
# HACK!
KODIR=/lib/modules/zing-driver/extra
KOBASE=zing_mm
KO=$KOBASE.ko
KOFILE=$KODIR/$KO

$ grep -i zing sos_commands/kernel/modinfo_ip6_tables_ebtable_nat_ebtables_ipt_MASQUERADE_iptable_n 
author:ERROR: modinfo: could not find module zing_mm
         ^     
          '....

$ cat etc/udev/rules.d/99-zing-md.rules
KERNEL=="zing_mm", MODE="0666", OWNER="root", GROUP="root"

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.