[RHEL] How do I check for hugepages usage and what is using it?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 8

Issue

  • How do I check for what is using my system hugepages?
  • How to caculate the size of hugepage used by a specified process?

Resolution

Kernel perspective

  1. First, check for the current hugepages usage:

    [root@server ~]# grep HugePages_ /proc/meminfo
    HugePages_Total:   16299
    HugePages_Free:     7764
    HugePages_Rsvd:     5330
    HugePages_Surp:        0
    
  2. Now, let's deduct the free pages, so we can find the used pages and sum to it the reserved pages. So the allocated pages is going to be Total - Free + Rsvd

    16299 - 7764 + 5330 = 13865
    
  3. Multiply the alloocated pages by 2048 (2048 is the number of kbytes of a single page)

    13865 x 2048 = 28,395,520 kbytes
    
  4. If you want to put the kbytes in byte count, multiply it by 1024 (1 kbyte = 1024 bytes):

    28395520 x 1024 = 29,077,012,480 bytes
    

Where is the memory

  • You can easily quantify the shared hugepages memory. To do so:
  1. List the shared memory in use:

    [root@rfreire sys]# ipcs -m
    ------ Shared Memory Segments --------
    key        shmid      owner      perms      bytes      nattch     status      
    0x00000000 163840     oracle     640        14680064   50                      
    0x00000000 196609     oracle     640        2499805184 50                      
    0x27126a4c 229378     oracle     640        2097152    50                      
    0x00000000 5636099    oracle     640        33554432   58                      
    0x00000000 5668868    oracle     640        4160749568 58                      
    [...]
    
  2. Sum the 5th column (quick shell: ipcs -m|awk '{ print $5}'|awk '{a+=$0}END{print a}') and match against /proc/meminfo hugepages information (see procedure above). If the values matches, then you have a hugepages-only shared memory. If you get a larger value, you have 4kb regular pages shared memory in use as well. Just deduct this sum from /proc/meminfo hugepages value in bytes and then you'll find how much you have of regular 4kb pages.

  • Check if you are using a hugepages filesystem. Grep for huge in /proc/mounts:

    [root@server ~]# grep -i huge /proc/mounts
    none /hugepages hugetlbfs rw,relatime 0 0
    
  • Unfortunately, at this time there are no means to quantify private hugetlbfs pages, which are used for qemu-kvm, for example.

How to caculate the size of hugepage used by a specified process

  • The following command can be used to caculate the size of hugepage used by a specified process, assumption that HugePage size is 2048 kB, the output unit is MiB:
grep -B 11 'KernelPageSize:     2048 kB' /proc/[PID]/smaps | grep "^Size:" | awk 'BEGIN{sum=0}{sum+=$2}END{print sum/1024}'
  • Note: avoid double counting of the same address in /proc/[PID]/smaps.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

1 Comments

I poked around in /proc/vm/$pid/numa_stat and found some info that appears to be hugepage usage.

I came up with this:

function pshugepage () {
HUGEPAGECOUNT=0
for num in grep 'anon_hugepage.*dirty=' /proc/$@/numa_maps | awk '{print $6}' | sed 's/dirty=//' ; do
HUGEPAGECOUNT=$((HUGEPAGECOUNT+num))
done
echo process $@ using $HUGEPAGECOUNT huge pages
}

I can't swear it is correct, but the numbers it gives are plausible.