Chapter 6. Resource Isolation and Tuning

The above Heat environment file has the following parameter:

OSD_NUMA_INTERFACE: 'em2'

Set the above to the name of the network device on which the Ceph services listen. In this reference architecture the device is em2, but the value may be determined for all deployments by either the StorageNetwork variable, or the StorageMgmtNetwork variable that was set in the Section 5.2, “Network Configuration” section. Workloads that are read-heavy benefit from using the StorageNetwork variable, while workloads that are write-heavy benefit from using the StorageMgmtNetwork variable. In this reference architecture both networks are VLANs on the same interface.

The following section of custom-templates/post-deploy-template.yaml contains a Heat config line and then embeds a shell script:

      config: |
        #!/usr/bin/env bash
        {
        echo "TODO: pin OSDs to the NUMA node of $OSD_NUMA_INTERFACE"
        } 2>&1 > /root/post_deploy_heat_output.txt

Update the above so that rather than embed a simple shell script, it instead includes a more complex shell script in a seprate file using Heat’s get_file intrinsic function.

      config: {get_file: numa-systemd-osd.sh}

The above change calls the script numa-systemd-osd.sh, which takes the network interface used for Ceph network traffic as an argument, and then uses lstopo to determine that interfaces’s NUMA node. It then modifies the systemd unit file for the Ceph OSD service so that numactl is used to start the OSD service with a NUMA policy that prefers the NUMA node of the Ceph network’s interface. It then restarts each Ceph OSD daemon sequentially so that the service runs with the new NUMA option.

When numa-systemd-osd.sh is run directly on a osd_compute node (with OSD_NUMA_INTERFACE set within the shell script), its output looks like the following:

[root@overcloud-osd-compute-0 ~]# ./numa-systemd-osd.sh
changed: --set /usr/lib/systemd/system/ceph-osd@.service Service ExecStart '/usr/bin/numactl -N 0 --preferred=0 /usr/bin/ceph-osd -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph'

Status of OSD 1 before unit file update

* ceph-osd@1.service - Ceph object storage daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-12-16 02:50:02 UTC; 11min ago
 Main PID: 83488 (ceph-osd)
   CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@1.service
           └─83488 /usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup ceph

Dec 16 02:50:01 overcloud-osd-compute-0.localdomain systemd[1]: Starting Ceph object storage daemon...
Dec 16 02:50:02 overcloud-osd-compute-0.localdomain ceph-osd-prestart.sh[83437]: create-or-move update...
Dec 16 02:50:02 overcloud-osd-compute-0.localdomain systemd[1]: Started Ceph object storage daemon.
Dec 16 02:50:02 overcloud-osd-compute-0.localdomain numactl[83488]: starting osd.1 at :/0 osd_data /v...l
Dec 16 02:50:02 overcloud-osd-compute-0.localdomain numactl[83488]: 2016-12-16 02:50:02.544592 7fecba...}
Dec 16 03:01:19 overcloud-osd-compute-0.localdomain systemd[1]: [/usr/lib/systemd/system/ceph-osd@.s...e'
Hint: Some lines were ellipsized, use -l to show in full.

Restarting OSD 1...

Status of OSD 1 after unit file update

* ceph-osd@1.service - Ceph object storage daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-12-16 03:01:21 UTC; 7ms ago
  Process: 89472 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i (code=exited, status=0/SUCCESS)
 Main PID: 89521 (numactl)
   CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@1.service
           └─89521 /usr/bin/numactl -N 0 --preferred=0 /usr/bin/ceph-osd -f --cluster ceph --id 1 --se...

Dec 16 03:01:21 overcloud-osd-compute-0.localdomain systemd[1]: Starting Ceph object storage daemon...
Dec 16 03:01:21 overcloud-osd-compute-0.localdomain ceph-osd-prestart.sh[89472]: create-or-move update...
Dec 16 03:01:21 overcloud-osd-compute-0.localdomain systemd[1]: Started Ceph object storage daemon.
Hint: Some lines were ellipsized, use -l to show in full.

Status of OSD 11 before unit file update
...

The logs of the node should indicate that numactl was used to start the OSD service.

[root@overcloud-osd-compute-0 ~]# journalctl | grep numa | grep starting
Dec 16 02:50:02 overcloud-osd-compute-0.localdomain numactl[83488]: starting osd.1 at :/0 osd_data /var/lib/ceph/osd/ceph-1 /var/lib/ceph/osd/ceph-1/journal
...

When the modified post-deploy-template.yaml and numa-systemd-osd.sh described above are run in Chapter 7, Deployment, the Ceph OSD daemons on each osd-compute node will be restarted under a NUMA policy. To verify that this has been completed after the deployment, check the log with journalctl as shown above or check the output of numa-systemd-osd.sh captured in /root/post_deploy_heat_output.txt.

The full content of post-deploy-template.yaml and numa-systemd-osd.sh may be read in the Appendix on Appendix D, Custom Heat Templates and is also available online as described in the Appendix G, GitHub Repository of Example Files Appendix for more details.

The numa-systemd-osd.sh script checks if the hwloc and numactl packages are installed, and if they are not installed, tries to install them with yum. To ensure these packages are available, consider either of the following options:

The numactl package is necessary so that the Ceph OSD processes can be started with a NUMA policy. The hwloc package provides the lstopo-no-graphics command, which shows the CPU topology of the system. Rather than require the user to determine which NUMA socket Ceph should be pinned to, based on the IRQ of the $OSD_NUMA_INTERFACE, the following exmaines the system to determine the desired NUMA socket number. It uses the lstopo-no-graphics command, filters the output with grep and then loops through the output to determine which NUMA socket has the IRQ.

declare -A NUMASOCKET
while read TYPE SOCKET_NUM NIC ; do
    if [[ "$TYPE" == "NUMANode" ]]; then
	NUMASOCKET=$(echo $SOCKET_NUM | sed s/L//g);
    fi
    if [[ "$NIC" == "$OSD_NUMA_INTERFACE" ]]; then
	# because $NIC is the $OSD_NUMA_INTERFACE,
	# the NUMASOCKET has been set correctly above
	break # so stop looking
    fi
done < <(lstopo-no-graphics | tr -d [:punct:] | egrep "NUMANode|$OSD_NUMA_INTERFACE")

The tr command is used to trim away punctuation, as lstopo-no-graphics outputs the network interface in quotes. A regular expression passed to egrep shows only the lines containing the NUMANode or the $OSD_NUMA_INTERFACE defined earlier. A while loop with read is used to extract the three columns containing the desired strings. Each NUMA socket number is collected, without the preceding 'L' as per sed. The $NUMASOCKET is set for each iteration containing the NUMAnode in case, during the next iteration, the $OSD_NUMA_INTERFACE is found. When the desired network interface is found, the loop exits with break before the $NUMASOCKET variable can be set to the next NUMA socket number. If no $NUMASOCKET is found, then the script exits.

The crudini command is used to save the ExecStart section of the default OSD unit file.

CMD=$(crudini --get $UNIT Service ExecStart)

A different crudini command is then used to put the same command back for the ExecStart command, but the command has a numactl call appended to its front.

crudini --verbose --set $UNIT Service ExecStart "$NUMA $CMD"

The $NUMA variable saves the numactl call to start the OSD daemon with a NUMA policy to only execute the command on the CPUs of the $NUMASOCKET identified previously. The --preferred option and not --membind is used. This is done because testing shows that hard pinning, with --membind, can cause swapping.

NUMA="/usr/bin/numactl -N $NUMASOCKET --preferred=$NUMASOCKET"

The last thing that numa-systemd-osd.sh does is to restart all of the OSD daemons on the server.

OSD_IDS=$(ls /var/lib/ceph/osd | awk 'BEGIN { FS = "-" } ; { print $2 }')
for OSD_ID in $OSD_IDS; do
  systemctl restart ceph-osd@$OSD_ID
done

A variation of the command above is used to show the status before and after the restart. This status is saved in /root/post_deploy_heat_output.txt on each osd-compute node.