How to use cgroup cpusets with systemd in RHEL7?

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux 7

Issue

  • How to use cgroup cpuset with systemd for user processes?

In the environment we have assigned exclusive access to individual CPU cores to individual, specific user processes/threads. Under RHEL 6.5, this was done using libcgroup and the cgruoup cpuset functionality (specifically via the cgconfig service and cgclassify). A one-time configuration script generates the cgconfig.conf and this is installed by a system administrator.

They are required to launch, then classify as the application makes use of LD_LIBRARY_PATH which is not available when running a binary with setuid (such as cgexec). However, these libcgroup tools have been subsequently deprecated in RHEL 7 with the following warning:

WARNING
The deprecated cgconfig tool from the libcgroup package is available to mount and handle hierarchies for controllers not yet supported by systemd (most notably the net-prio controller). Never use libcgropup tools to modify the default hierarchies mounted by systemd since it would lead to unexpected behavior. The libcgroup library will be removed in the future versions of Red Hat Enterprise Linux. For more information on how to use cgconfig, see Chapter 3, Using libcgroup Tools.

The replacement model uses systemd as described here

However, currently the cpuset interface is not exposed through system and, so as far as we can tell, this functionality is now not available: [2][3]
Note that the number of cgroup attributes currently exposed as unit properties is limited. This will be extended later on, as their kernel interfaces are cleaned up. For example cpuset or freezer are currently not exposed at all due to the broken inheritance semantics of the kernel logic. Also, migrating units to a different slice at runtime is not supported (i.e. altering the Slice= property for running units) as the kernel currently lacks atomic cgroup subtree moves.

How is it possible to safely achieve the same result (exclusive cpu access/arbitrary cpusets for userland applications) ?

Resolution

  • As per RHEL 7 product documentation libcgroup is deprecated and systemd provides some directives which is helpful to use the cgroup feature (As of now systemd does not provide directives for all cgroup feature)

  • Specific to limit cgroup resources for those systemd does not provide directive i am sharing one dummy example which confirm how we can use systemd to configure cgroup resources.

    //cc cpu.c -o cpu 
    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>
    int main()
    {
            double x;
            while(1)
            {
                    x = x + 0.0000001;
            }
            return 0;
    }
    
  • Below is the service file to run this program through the systemd and configure cgroup resource limit for this program

    # cat /usr/lib/systemd/system/cpus.service
    [Unit]
    Description=cpus
    After=syslog.target network.target auditd.service
    
    [Service]
    ExecStartPre=/usr/bin/mkdir -p /sys/fs/cgroup/cpuset/mygroup1 ===> Create group to manage process 
    ExecStartPre=/bin/bash -c '/usr/bin/echo "3" > /sys/fs/cgroup/cpuset/mygroup1/cpuset.cpus' ==> Assign cpu core 3 to this process
    ExecStartPre=/bin/bash -c '/usr/bin/echo "0" > /sys/fs/cgroup/cpuset/mygroup1/cpuset.mems'
    ExecStart=/root/cpu ===> Run this process
    ExecStartPost=/bin/bash -c '/usr/bin/echo $MAINPID > /sys/fs/cgroup/cpuset/mygroup1/tasks' ==> Assign process id to group task file
    ExecStopPost=/usr/bin/rmdir /sys/fs/cgroup/cpuset/mygroup1 ==> At the time of stop remove group
    Restart=on-failure
    
    [Install]
    WantedBy=multi-user.target
    
  • After creating the file, execute the service

    # systemctl daemon-reload
    # systemctl start cpus.service
    
  • It will start cpu process and upon investigation should be clear it is using CPU 3.

    # ps -eLF | grep -e PID -e cpu
    UID        PID  PPID   LWP  C NLWP    SZ   RSS PSR STIME TTY          TIME CMD
    root     131188      1 131188 98    1  1037   320   3 17:21 ?        00:08:59 /root/cpu
    
  • To run in a unprivileged user we need to divide the unit file into two separate unit files.

    1. The commands which needs root privileges put them into the first unit1.service file

      [Service]
      ExecStartPre=/usr/bin/mkdir -p /sys/fs/cgroup/cpuset/mygroup1 ===> Create group to manage process 
      ExecStartPre=/bin/bash -c '/usr/bin/echo "3" > /sys/fs/cgroup/cpuset/mygroup1/cpuset.cpus' ==> Assign cpu core 3 to this process
      ExecStartPre=/bin/bash -c '/usr/bin/echo "0" > /sys/fs/cgroup/cpuset/mygroup1/cpuset.mems'
      ExecStartPost=/bin/bash -c '/usr/bin/echo $MAINPID > /sys/fs/cgroup/cpuset/mygroup1/tasks' ==> Assign process id to group task file
      ExecStopPost=/usr/bin/rmdir /sys/fs/cgroup/cpuset/mygroup1 ==> At the time of stop remove group
      
    2. And then put command/binary which does not need root privileges put them into another unit2.service file

      [Service]
      ExecStart=/root/cpu ===> Run this process
      Restart=on-failure
      

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments