Unaccounted memory usage when running Red Hat Enterprise Linux with ZFS filesystem

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux [ALL]
  • ZFS filesystems mounted

Issue

  • What is consuming all of the memory?
  • Unable to account for memory usage in the RHEL system when ZFS filesystem is used.
# free_-m 
             total       used       free     shared    buffers     cached
Mem:       2067485    2063584       3901        553        542     672621
-/+ buffers/cache:    1390420     677064
Swap:      2097147          0    2097147

# cat proc/cmdline 
ro root=/dev/mapper/vg0-lv_root nomodeset rd_NO_LUKS rd_LVM_LV=vg0/lv_root LANG=C rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=257M@0M rd_LVM_LV=vg1/vg_swap  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb transparent_hugepage=never quiet
  • System shows high memory usage even though applications are not allocating that much memory :
# cat ps |tail -n +2 | cut -b2- | gawk '{mem += $6} END {print "Total RSS Mem :" mem/1048576 "GB"}'
Total RSS Mem :302.007GB
  • There are no Hugepages configured in the system

Resolution

  • ZFS filesystem maintains temporary files (similar to OS cache) to increase performance in different kernel data structures which is not reported in normal system tools like free, top, etc.
  • ZFS filesystem is not supported by Red Hat.
  • Memory usage for the ZFS can be checked with the arcstat.py script or proc filesystem as below.
/usr/bin/arcstat.py 10 10
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c
20:21:23   136   114     83    92   80    21   99     0    2   889G  889G
20:21:33    77    62     81    47   76    15  100     0    1   889G  889G
20:21:43   166   121     73    87   66    34  100     0    2   889G  889G
20:21:53    86    58     67    49   63     9  100     0    5   889G  889G
                                                                ^^^ 

# cat /proc/spl/kstat/zfs/arcstats
c                               4    957050822568
c_min                           4    4194304
c_max                           4    1083957981184
size                            4    957050737752
size                            4    957050737752

Root Cause

  • The ARC grows and consumes memory on the principle that no need exists to return data to the system while there is still plenty of free memory. When the ARC has grown and outside memory pressure exists, for example, when a new application starts up, then the ARC releases its hold on memory.

  • Memory consumed by ZFS ARC cache would not be represented by normal system tools like free and top, we can check proc filesystem as above to get memory consumed by ZFS cache.

Diagnostic Steps

  • Verify Server does not have Hugepages configured. If Hugepages are configured then there might be some RAM accounted as used under free, but in real it is just been reserved part of RAM.
    Eg:
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
  • Verify overall system memory usage from the output of free command.
    Eg:
# cat free 
             total       used       free     shared    buffers     cached
Mem:    2117105432 2113111144    3994288     566448     555136  688764256
-/+ buffers/cache: 1423791752  693313680
Swap:   2147479548          0 2147479548
  • Verify total memory utilization of process(es) running on system from the output of "ps" command output.
    Eg:
# cat ps |tail -n +2 | cut -b2- | gawk '{mem += $6} END {print "Total RSS Mem :" mem/1048576 "GB"}'
Total RSS Mem :302.007GB
  • Check VMware Memory Control Driver ( vmmemctl.ko or vmware_balloon.ko ) details.
    Eg:
# modinfo vmmemctl
or
# modinfo vmware_balloon
  • Check ZFS module details as well as mount details details of the .partitions.
    Eg:
# cat sos_commands/kernel/modinfo_nfs_lockd_fscache_auth_rpcgss_nfs_acl_sunrpc_lin_tape_cp |grep zfs -i -A 10
filename:       /lib/modules/2.6.32-504.16.2.el6.x86_64/extra/zfs.ko
version:        0.6.4.1-1
license:        CDDL
author:         OpenZFS on Linux
description:    ZFS
srcversion:     8324F6AEA2A06B2B6F0A0F5
depends:        spl,znvpair,zcommon,zunicode,zavl
vermagic:       2.6.32-504.16.2.el6.x86_64 SMP mod_unload modversions

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.