Systems running relatively short-lived processes which heavily utilize memory and processors experience slowdowns on some NUMA nodes

Solution Verified - Updated -

Environment

  • Bull bullion S16 system
  • Red Hat Enterprise Linux 7
  • Intel Xeon E7 v4-series processors

Issue

Red Hat has observed varying levels of performance between NUMA nodes when systems run workloads similar to those described in the title with transparent hugepages enabled. Some nodes were operating approximately 10% slower than others during testing. The cause was traced to transparent hugepage scanning and hugepage conversion, and disabling transparent hugepages brought performance back within normal tolerances across the nodes.

Resolution

If you are experiencing this problem, Red Hat recommends disabling transparent hugepages to ensure consistent performance across all NUMA nodes.

To disable transparent hugepages until the next reboot, run the following commands as root:

# echo never > /sys/kernel/mm/transparent_hugepage/enabled
# echo never > /sys/kernel/mm/transparent_hugepage/defrag

Note that processes already converted to huge pages will remain until they exit.

To permanently disable transparent hugepages at boot time, two options are available:

Option one: Modifying kernel boot arguments
Add "transparent_hugepage=never" to the kernel boot arguments using the grubby command, then reboot your system:

1) As root, determine the kernel(s) you wish to modify to disable transparent hugepages:

# grubby --info=ALL

[Sample output from a Red Hat Enterprise Linux 7.2 system appears below]

index=0
kernel=/boot/vmlinuz-3.10.0-327.36.1.el7.x86_64
args="ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet LANG=en_US.UTF-8"
root=/dev/mapper/rhel-root
initrd=/boot/initramfs-3.10.0-327.36.1.el7.x86_64.img
title=Red Hat Enterprise Linux Server (3.10.0-327.36.1.el7.x86_64) 7.2 (Maipo)
index=1
kernel=/boot/vmlinuz-3.10.0-327.28.3.el7.x86_64
args="ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet LANG=en_US.UTF-8"
root=/dev/mapper/rhel-root
initrd=/boot/initramfs-3.10.0-327.28.3.el7.x86_64.img
title=Red Hat Enterprise Linux Server (3.10.0-327.28.3.el7.x86_64) 7.2 (Maipo)
...

Make a note of the "kernel=" value for the kernel(s) you wish to update.

2) Add the necessary argument to the kernel or kernels you wish to update:

# grubby --args="transparent_hugepage=never" --update-kernel $KERNEL

Replace "$KERNEL" with the "kernel=" value from the output of the first grubby command for each kernel you wish to update. Only one kernel section at a time can be updated, so the command will need to be repeated if you wish to modify multiple kernels. As an example, "--update-kernel /boot/vmlinuz-3.10.0-327.36.1.el7.x86_64" would modify the argument for the kernel listed under index 0 in the above example.

Should you wish to remove the argument, use the "--remove-args=" option as shown below, again replacing "$KERNEL" with the appropriate kernel from the output of the grubby --info=ALL command.

# grubby --remove-args="transparent_hugepage=never" --update-kernel $KERNEL

3) Reboot once you have completed your updates to allow the change to take effect.

For more information on using the grubby command, please see Section 24.4 of the Red Hat Enterprise Linux System Administrator's Guide entitled "Making Persistent Changes to a GRUB 2 Menu Using the Grubby Tool", available at the following link on the Red Hat Customer Portal:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/sec-Making_Persistent_Changes_to_a_GRUB_2_Menu_Using_the_grubby_Tool.html

Option two: tuned profiles
Another method for controlling transparent hugepages and other performance-related parameters is to use tuned, the system tuning daemon. Through the use of custom or predefined profiles, tuned can enable or disable transparent hugepages and make other changes to memory, CPU, and additional subsystems to tune system performance appropriately for your application.

Note: tuned is active by default on Red Hat Enterprise Linux 7. Changing tuned profiles while the system is running can override changes to /sys parameters you may have made using the echo command, but they will not override the boot line argument that controls transparent hugepages as explained in option one above.

There are many ways in which tuned could be used to disable transparent hugepages. The example below references a user-created tuned profile called "disable-transparent-hugepages" which includes all the settings from the existing "throughput-performance" profile and adds the necessary setting to disable transparent hugepages:

# cat /usr/lib/tuned/disable-transparent-hugepages/tuned.conf

#
# tuned configuration to disable transparent hugepages
#

[main]
include=throughput-performance

[vm]
transparent_hugepages=never

Running the following command will enable the new profile and its settings immediately and keep those settings active across reboots:

# tuned-adm profile disable-transparent-hugepages

For more information on tuned and tuning, please see Section 2.6 of the Red Hat Enterprise Linux 7 Performance Tuning Guide entitled "Tuned and Tuned-adm", available at the following link on the Red Hat Customer Portal:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7-Beta/html/Performance_Tuning_Guide/sect-Red_Hat_Enterprise_Linux-Performance_Tuning_Guide-Performance_Monitoring_Tools-tuned_and_tuned_adm.html

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments