[RHEL] How do I check for hugepages usage and what is using it?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 9

Issue

  • How do I check for what is using my system hugepages?
  • How to calculate the size of hugepage used by a specified process?

Resolution

Kernel perspective

  1. First, check for the current hugepages usage:

    [root@server ~]# grep -i hugepages /proc/meminfo
HugePages_Total:   16299
HugePages_Free:     7764
HugePages_Rsvd:     5330
HugePages_Surp:        0
Hugepagesize:       2048 kB

  2. Now, let's deduct the free pages, so we can find the used pages and sum to it the reserved pages. So the allocated pages is going to be Total - Free + Rsvd:

    16299 - 7764 + 5330 = 13865

  3. Multiply the allocated pages by 2048 (The size of hugepage in kbytes):

    13865 x 2048 = 28,395,520 kbytes

  4. If you want to put the kbytes in byte count, multiply it by 1024 (1 kbyte = 1024 bytes):

    28395520 x 1024 = 29,077,012,480 bytes

Where is the memory?

  • You can easily quantify the shared hugepages memory. To do so:

  1. List the shared memory in use:

    [root@rfreire sys]# ipcs -m
------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x00000000 163840     oracle     640        14680064   50                      
0x00000000 196609     oracle     640        2499805184 50                      
0x27126a4c 229378     oracle     640        2097152    50                      
0x00000000 5636099    oracle     640        33554432   58                      
0x00000000 5668868    oracle     640        4160749568 58                      
[...]

  2. Sum the 5th column (quick shell: ipcs -m|awk '{ print $5}'|awk '{a+=$0}END{print a}') and match against /proc/meminfo hugepages information (see procedure above). If the values matches, then you have a hugepages-only shared memory. If you get a larger value, you have 4kb regular pages shared memory in use as well. Just deduct this sum from /proc/meminfo hugepages value in bytes and then you'll find how much you have of regular 4kb pages.

  • Check if you are using a hugepages filesystem. Grep for huge in /proc/mounts:

    [root@server ~]# grep -i huge /proc/mounts
none /hugepages hugetlbfs rw,relatime 0 0

  • Unfortunately, at this time there are no means to quantify private hugetlbfs pages, which are used for qemu-kvm, for example.

How to calculate the size of hugepage used by a specified process?

[Step1] Find the process[s] that are currently utilizing the hugepages. You can skip this step if you are aware of the PID utilizing hugepages and move to step2.

grep "KernelPageSize:     2048 kB" /proc/[[:digit:]]*/smaps | awk {'print $1'} | cut -d "/" -f3 | sort | uniq

[Step2] Use the following command to calculate the size of the hugepage used by a specified process, assumption that HugePage size is 2048 kB, the output unit is MiB:

grep -B 11 'KernelPageSize:     2048 kB' /proc/[PID]/smaps | grep "^Size:" | awk 'BEGIN{sum=0}{sum+=$2}END{print sum/1024}'
  • Note: Avoid double counting of the same address in /proc/[PID]/smaps.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments