Chapter 3. Realtime-Specific Tuning

3.1. Setting Scheduler Priorities
3.2. MRG Realtime Specific gettimeofday speedup
3.3. Using kdump and kexec with the MRG Realtime kernel
3.4. TSC timer synchronization on Opteron CPUs
3.5. Infiniband
3.6. Non-Uniform Memory Access
3.7. Using the ftrace Utility for Tracing Latencies
3.8. Latency Tracing Using trace-cmd
3.9. Using sched_nr_migrate to limit SCHED_OTHER task migration.
Once you have completed the tunes in Chapter 2, General System Tuning you are ready to start MRG Realtime specific tuning. You must have the MRG Realtime kernel installed for these procedures.

Important

Do not attempt to use the tools in this section without first having completed Chapter 2, General System Tuning. You will not see a performance improvement.
When are you ready to begin MRG Realtime tuning, perform these steps first, as they will provide the greatest benefit:
When you are ready to start some fine-tuning on your system, then try the other sections in this chapter:
This chapter also includes information on performance monitoring tools:
When you have completed all the tuning suggestions in this chapter, move on to Chapter 4, Application Tuning and Deployment

3.1. Setting Scheduler Priorities

The MRG Realtime kernel allows fine grained control of scheduler priorities. It also allows application level programs to be scheduled at a higher priority than kernel threads. This can be useful but may also carry consequences. It is possible that it will cause the system to hang and other unpredictable behavior if crucial kernel processes are prevented from running as needed. Ultimately the correct settings are workload dependent.
Priorities are defined in groups, with some groups dedicated to certain kernel functions:

Table 3.1. Priority Map

Priority Threads Description
1 Low priority kernel threads Priority 1 is usually reserved for those tasks that need to be just above SCHED_OTHER
2 - 69 Available for use Range used for typical application priorities
70 - 79 Soft IRQs
80 NFS RPC, Locking and Authentication threads for NFS
81 - 89 Hard IRQs Dedicated interrupt processing threads for each IRQ in the system
90 - 98 Available for use For use only by very high priority application threads
99 Watchdogs and migration System threads that must run at the highest priority

Using rtctl to Set Priorities

  1. Priorities are set using a series of levels, ranging from 0 (lowest priority) to 99 (highest priority). The system startup script rtctl initializes the default priorities of the kernel threads. By requesting the status of the rtctl service, you can view the priorities of the various kernel threads. 
    # service rtctl status
2   TS      - [kthreadd]
3   FF     99 [migration/0]
4   FF     99 [posix_cpu_timer]
5   FF     50 [softirq-high/0]
6   FF     50 [softirq-timer/0]
7   FF     90 [softirq-net-tx/]
...[output truncated]...
  2. The output is in the format: 
    [PID] [scheduler policy] [priority] [process name]
    In the scheduler policy field, a value of TS indicates a policy of normal and FF indicates a policy of FIFO (first in, first out).
  3. The rtctl system startup script relies on the /etc/rtgroups file. To make changes, open the /etc/rtgroups file in your preferred text editor. 
    kthreads:*:1:*:\[.*\]$
watchdog:f:99:*:\[watchdog.*\]
migration:f:99:*:\[migration\/.*\]
softirq:f:70:*:\[.*(softirq|sirq).*\]
softirq-net-tx:f:75:*:\[(softirq|sirq)-net-tx.*\]
softirq-net-rx:f:75:*:\[(softirq|sirq)-net-rx.*\]
softirq-sched:f:1:*:\[(softirq|sirq)-sched\/.*\]
rpciod:f:65:*:\[rpciod.*\]
lockd:f:65:*:\[lockd.*\]
nfsd:f:65:*:\[nfsd.*\]
hardirq:f:85:*:\[(irq|IRQ)[\-_/].*\]
  4. Each line represents a process. You can change the priority of the process by adjusting the parameters. The entries in this file are in the format: 
    [group name]:[scheduler policy]:[scheduler priority]:[regular expression]
    In the scheduler policy field, the following values are accepted:
    o Sets a policy of other. If the policy is set to o, the scheduler priority field will be set to 0 and ignored.
    b Sets a policy of batch.
    f Sets a policy of FIFO.
    * If the policy is set to *, no change will be made to to any matched thread policy.
    The regular expression field matches the thread name to be modified.
  5. After editing the file, you will need to restart the rtctl service to reload it with the new settings: 
    # service rtctl stop

# service rtctl start
Setting kernel thread priorities: done
Related Manual Pages
For more information, or for further reading, the following man pages are related to the information given in this section.
  • rtctl(1)
  • rtgroups(5)