Installing RHEL 9 for Real Time

Red Hat Enterprise Linux for Real Time 9

Installing the Linux real-time kernel on Red Hat Enterprise Linux 9

Red Hat Customer Content Services

Abstract

This guide describes how to install and get started with the low-latency real-time kernel in RHEL 9.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.

Providing feedback on Red Hat documentation

We appreciate your feedback on our documentation. Let us know how we can improve it.

Submitting comments on specific passages

  1. View the documentation in the Multi-page HTML format and ensure that you see the Feedback button in the upper right corner after the page fully loads.
  2. Use your cursor to highlight the part of the text that you want to comment on.
  3. Click the Add Feedback button that appears near the highlighted text.
  4. Add your feedback and click Submit.

Submitting feedback through Bugzilla (account required)

  1. Log in to the Bugzilla website.
  2. Select the correct version from the Version menu.
  3. Enter a descriptive title in the Summary field.
  4. Enter your suggestion for improvement in the Description field. Include links to the relevant parts of the documentation.
  5. Click Submit Bug.

Chapter 1. Installing RHEL for Real Time

Many industries and organizations need extremely high performance computing and may require low and predictable latency, especially in the financial and telecommunications industries. Latency, or response time, is defined as the time between an event and system response and is generally measured in microseconds (μs).

For most applications running under a Linux environment, basic performance tuning can improve latency sufficiently. For those industries where latency not only needs to be low, but also accountable and predictable, Red Hat developed a 'drop-in' kernel replacement that provides this. RHEL for Real Time is distributed as part of RHEL 9 and provides seamless integration with RHEL 9. RHEL for Real Time offers clients the opportunity to measure, configure, and record latency times within their organization.

Warning

Before installing RHEL for Real Time, ensure that the base platform is properly tuned and the system BIOS parameters are adjusted. Failure to perform these tasks may prevent getting consistent performance from a RHEL Real Time deployment.

1.1. RHEL for Real Time for optimizing latency

RHEL for Real Time is designed to be used on well-tuned systems for applications with extremely high determinism requirements. Kernel system tuning offers the vast majority of the improvement in determinism.

For example, in many workloads, thorough system tuning improves consistency of results by around 90%. This is why, before using RHEL for Real Time, we recommend that customers first perform system tuning of standard RHEL to see if it meets their objectives.

System tuning is just as important when using the Real Time kernel as it is for the standard kernel. Installing the Real Time kernel on an untuned system running the standard kernel supplied as part of the RHEL 9 release is not likely to result in any noticeable benefit. Tuning the standard kernel will yield 90% of the possible latency gains. The Real Time kernel provides the last 10% of latency reduction required by the most demanding workloads.

Warning

Before tuning Real Time kernel systems, ensure that the base platform is properly tuned and the system BIOS parameters are adjusted. Failure to perform these tasks may prevent getting consistent performance from a RHEL Real Time deployment.

The objective of the Real Time kernel consistent, low-latency determinism offering predictable response times. There is some additional kernel overhead associated with the real time kernel. This is due primarily to handling hardware interrupts in separately scheduled threads. The increased overhead in some workloads results in some degradation in overall throughput. The exact amount is very workload dependent, ranging from 0% to 30%.

For typical workloads with kernel latency requirements in the millisecond (ms) range, the standard RHEL 9 kernel is sufficient. However,if your workload has stringent low-latency determinism requirements for core kernel features such as interrupt handling and process scheduling in the microsecond (μs) range, then the Real Time kernel is for you.

Benefit of realtime over standard kernel system tuning

This graph compares a million samples of machines that use the RHEL 9 and the RHEL for Real Time kernel, respectively.

  • The blue points in this graph represent the system response time (in microseconds) of machines running a tuned RHEL 9 kernel.
  • The green points in the graph represent the system response time of machines running a tuned real-time kernel.

It is clear from this graph that the response time of the Real Time kernel is very consistent, in contrast to the standard kernel, which has greater variability.

1.2. Installing RHEL for Real-Time using yum

To install RHEL 9 for Real Time on your system, you can use the yum package manager.

Note

An ISO image containing RHEL for Real Time is also available for download from the Customer Portal. You can use this ISO image to obtain all the RPM packages included with RHEL for Real Time. However, because this is not a bootable ISO image, you cannot use it to create a bootable USB or CD medium.

Prerequisites

  • The latest version of RHEL 9 is installed on an AMD64 or Intel64 system.

    Note

    RHEL for Real Time runs on AMD64 and Intel 64 (also known as x86_64) server platforms that are certified to run Red Hat Enterprise Linux.

  • Your machine is registered and RHEL is attached to a RHEL for Real Time subscription.

  • Ensure that the base platform is properly tuned and the system BIOS parameters are adjusted.

    Note

    Failure to perform these tasks may prevent getting consistent performance from a RHEL for Real Time deployment.

Procedure

  1. Enable the RHEL for Real Time repository. 

    # subscription-manager repos --enable rhel-9-for-x86_64-rt-rpms

  2. Install the RHEL for Real Time package group. 

    # yum groupinstall RT

    This group installs several packages:

    • kernel-rt - the RHEL for Real Time kernel package
    • kernel-rt-core - the core RHEL for Real Time kernel package
    • kernel-rt-devel - the RHEL for Real Time kernel development package
    • kernel-rt-modules - the RHEL for Real Time kernel modules package
    • kernel-rt-modules-extra - the RHEL for Real Time kernel extra modules package
    • realtime-setup- a package that sets up the basic environment required by RHEL for Real Time
    • rteval - a package that evaluates system suitability for RHEL for Real Time
    • rteval-loads - a package that provides source code for rteval loads
    • tuned-profiles-realtime - a package with additional TuneD profiles targeted to realtime

  3. (Optional) Additionally, the tuna package contains a tool that helps tune the RHEL for Real Time workload, greatly automating CPU isolation and thread affinity operations from the command line or the GUI. This package is available in the base RHEL 9 repository. For information on using tuna, see the RHEL for Real Time Kernel Tuning Guide. 

    # yum install tuna
Note

When the RHEL for Real Time kernel is installed, it is automatically set to be the default kernel and is used on the next boot.

Verification steps

  • Check the installation location and verify that the components have been installed successfully. 

    # rpm -ql realtime-setup
/etc/security/limits.d/realtime.conf
/etc/sysconfig/realtime-setup
/etc/udev/rules.d/99-rhel-rt.rules
/usr/bin/realtime-setup
/usr/bin/rt-setup-kdump
/usr/bin/slub_cpu_partial_off
/usr/lib/.build-id
/usr/lib/.build-id/a4
/usr/lib/.build-id/a4/da77908aa4c6f048939f3267f1c552c456d117
/usr/lib/systemd/system/rt-entsk.service
/usr/lib/systemd/system/rt-setup.service
/usr/sbin/kernel-is-rt
/usr/sbin/rt-entsk

Additional resources

1.3. Available RPM packages in the RHEL for Real Time repository

This section lists the RPM packages available in the RHEL for Real Time repository.

The RHEL for Real Time Specific column indicates if the RPM differs from the standard RHEL 9 maintenance stream or is not applicable to the standard kernel. A No in this column indicates that the RPM performs equivalently on RHEL 9.

The Required column indicates whether the package is mandatory for correct RHEL for Real Time behavior. A No in this column indicates that usage is optional.

Table 1.1. Basic RHEL for Real Time kernel packages

RPM package nameDescriptionRT-specificRequired

kernel-rt

Low latency and preemption functionality

Yes

Yes

The following packages contain test programs for use with RHEL for Real Time.

Table 1.2. RHEL for Real Time kernel test packages

RPM package nameDescriptionRT-specificRequired

kernel-rt-devel

Headers and libraries for kernel development

Yes

No

kernel-rt-debug

RHEL for Real Time kernel with debugging functions compiled in (slow)

Yes

No

kernel-rt-debug-devel

Headers and libraries for development on debug kernel

Yes

No

rt-tests

Utilities for measuring system latencies and for proving that priority-inheritance mutexes function properly

No

No

The following set of packages are provided for use with the perf, trace-cmd, and crash utilities for analyzing kernel crash dumps. The debugging packages consist of symbol tables and are quite large. For this reason, they are separately delivered from the other RHEL for Real Time packages.

These packages can be found and downloaded from the RHEL for Real Time - Debug RPMs repository.

Table 1.3. Basic RHEL for Real Time debugging packages

RPM package nameDescriptionRT-specificRequired

kernel-rt-debuginfo

Symbols for profiling and debugging use, such as perf or trace-cmd

Yes

No

kernel-rt-debug-debuginfo

Symbols for profiling and tracing

Yes

No

1.4. Post installation instructions

After you install RHEL for Real Time ensure that:

  • To achieve optimal low-latency determinism, you perform RHEL for Real Time specific system tuning.
  • You know about the module compatibility of the real time kernel and the standard kernel.
  • To enable kdump, you must configure RHEL for Real Time to provide crash dump information by enabling kexec/kdump.
  • Verify that the Real Time kernel is the default kernel.

Module Compatibility of the Real Time Kernel and the Standard Kernel

RHEL for Real Time differs substantially from the standard Red Hat Enterprise Linux 9 kernel. As a consequence, third-party kernel modules are incompatible with RHEL for Real Time.

Kernel modules are inherently specific to the kernel they are built for. The real time kernel is substantially different from the standard kernel, and so are the modules.

In other words, you cannot take third-party modules from Red Hat Enterprise Linux 9 and use them as-is on RHEL for Real Time. If you must use a third-party module, you must recompile it with the RHEL for Real Time header files, which are available in the RHEL for Real Time development and test packages.

The following are some example third-party drivers which ship for standard Red Hat Enterprise Linux 9 but do not currently have a custom build for RHEL for Real Time:

  • EMC Powerpath
  • NVidia graphics
  • Advanced storage adapter configuration utilities from Qlogic

The user space syscall interface is compatible with RHEL for Real Time.

Chapter 2. Reporting RHEL for Real Time bugs

Identifying which kernel, the RHEL for Real Time or the standard kernel, is causing the problem might increase the chances of getting your bug fixed. This section covers information on how to diagnose a bug and report it.

2.1. Diagnosing RHEL for Real Time bugs

This procedure guides you how to diagnose where the problem has occurred before you file a bug report.

Prerequisite:

  • The latest version of RHEL for Real Time kernel is installed.

Procedure:

  1. Verify that you have the latest version of the RHEL for Real Time kernel.
  2. Boot into RHEL for Real Time kernel using the GRUB menu.
  3. If the problem occurs, report a bug against RHEL for Real Time.

  4. Try to reproduce the problem with the standard kernel.

    This troubleshooting step might assist in identifying the problem location.

Note

If the problem does not occur with the standard kernel, then the bug is probably the result of changes introduced in the RHEL for Real Time specific enhancements, which Red Hat has applied on top of the baseline (4.18.0) kernel.

2.2. Submitting a bug report with Bugzilla

If you have determined that the bug is specific to Red Hat Enterprise Linux for Real Time follow the instructions in this section to submit a bug report:

Prerequisite:

  • You have a Red Hat Bugzilla account.

Procedure

  1. Log into your Bugzilla account.
  2. Click Enter A New Bug Report.
  3. Select Red Hat classification.
  4. Select the Red Hat Enterprise Linux product.
  5. Enter Component.
    For example, use kernel-rt if it is a kernel issue or the name of the affected user space component, such as rteval.

  6. Provide a detailed description of the bug issue for RHEL for Real Time kernel.

    When entering the problem description you might also want to state if you were able to reproduce the problem on the standard Red Hat Enterprise Linux 9 kernel.

Additional resources

Legal Notice

Copyright © 2022 Red Hat, Inc.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.