Chapter 3. Realtime-Specific Tuning

Once you have completed the optimization in Chapter 2, General System Tuning you are ready to start Red Hat Enterprise Linux for Real Time specific tuning. You must have the Red Hat Enterprise Linux for Real Time kernel installed for these procedures.

Important

Do not attempt to use the tools in this section without first having completed Chapter 2, General System Tuning. You will not see a performance improvement.
When are you ready to begin Red Hat Enterprise Linux for Real Time tuning, perform these steps first, as they will provide the greatest benefit:
When you are ready to start some fine-tuning on your system, then try the other sections in this chapter:
This chapter also includes information on performance monitoring tools:
When you have completed all the tuning suggestions in this chapter, move on to Chapter 4, Application Tuning and Deployment

3.1. Setting Scheduler Priorities

The Red Hat Enterprise Linux for Real Time kernel allows fine grained control of scheduler priorities. It also allows application level programs to be scheduled at a higher priority than kernel threads. This is useful but it can also carry consequences. It is possible that it will cause the system to hang and other unpredictable behavior if crucial kernel processes are prevented from running as needed. Ultimately the correct settings are workload dependent.
Priorities are defined in groups, with some groups dedicated to certain kernel functions:

Table 3.1. Priority Map

Priority Threads Description
1 Low priority kernel threads Priority 1 is usually reserved for those tasks that need to be just above SCHED_OTHER
2 - 49 Available for use Range used for typical application priorities
50 Default hard-IRQ value
51 - 98 High priority threads Use this range for threads that execute periodically and must have quick response times. Do not use this range for CPU-bound threads as you will starve interrupts.
99 Watchdogs and migration System threads that must run at the highest priority

Procedure 3.1. Using rtctl to Set Priorities

  1. Priorities are set using a series of levels, ranging from 0 (lowest priority) to 99 (highest priority). The system startup script, rtctl, may be used to change the default priorities of threads following kernel boot. Its default is to not modify any thread priorities.
    To view scheduling priorities of running threads, use the tuna utility: 
    ~]# tuna --show_threads
                      thread       ctxt_switches
    pid SCHED_ rtpri affinity voluntary nonvoluntary             cmd
  2      OTHER     0    0xfff       451            3        kthreadd  
  3       FIFO     1        0     46395            2     ksoftirqd/0  
  5      OTHER     0        0        11            1    kworker/0:0H  
  7       FIFO    99        0         9            1   posixcputmr/0 
...[output truncated]...
  2. The rtctl script relies on the /etc/rtgroups file: 
    ...[comments_truncated]...
# kthreads:*:*:*:\[.*\]$
# softirq:f:1:*:\[ksoftirqd\/.*\]
# rcu:f:1:*:\[rcu[bc]\/.*\]
# rpciod:o:*:*:\[rpciod.*\]
# lockd:*:*:*:\[lockd.*\]
# nfsd:*:*:*:\[nfsd.*\]
# hardirq:f:50:*:\[irq[\-_/].*\]
# rcun:f:99:*:\[rcun\/.*\]
# watchdog:f:99:*:\[watchdog\/.*\]
# migration:f:99:*:\[migration\/.*\]
  3. Each line defines a group that will match some set of threads. By default, all the groups are commented out with a leading hash sign (#). You can uncomment a group and edit its parameters, or you can add a new group with your own name and parameters. You can also change the priority of a group by adjusting its parameters.

    Warning

    Do not change the priority of the kernel threads because these threads are meant to run only when triggered. Changing their priority may have a negative impact.
    In fact, the migration thread is a stop thread and is protected in the sched_setscheduler() kernel function. Any attempt at changing the policy and priority of the migration/x thread (where x is the number of per-CPU core) will fail with an Invalid argument message.
    The entries in this file are written in the following format: 
    [group name]:[scheduler policy]:[scheduler priority]:[regular expression]
    In the scheduler policy field, the following values are accepted:
    o Sets a policy of other. If the policy is set to o, the scheduler priority field will be set to 0 and ignored.
    b Sets a policy of batch.
    f Sets a policy of FIFO.
    * If the policy is set to *, no change will be made to any matched thread policy.
    The regular expression field matches the thread name to be modified.
  4. After editing the file, you will need to restart the rtctl service to reload it with the new settings: 
    ~]# systemctl stop rtctl
~]# systemctl start rtctl
Related Manual Pages

For more information, or for further reading, the following man pages are related to the information given in this section.

  • rtctl(1)
  • rtgroups(5)