Red Hat Training

A Red Hat training course is available for RHEL 8

Chapter 35. Cross-instrumentation of SystemTap

Cross-instrumentation of SystemTap is creating SystemTap instrumentation modules from a SystemTap script on one system to be used on another system that does not have SystemTap fully deployed.

35.1. SystemTap cross-instrumentation

When you run a SystemTap script, a kernel module is built out of that script. SystemTap then loads the module into the kernel.

Normally, SystemTap scripts can run only on systems where SystemTap is deployed. To run SystemTap on ten systems, SystemTap needs to be deployed on all those systems. In some cases, this might be neither feasible nor desired. For example, corporate policy might prohibit you from installing packages that provide compilers or debug information on specific machines, which will prevent the deployment of SystemTap.

To work around this, use cross-instrumentation. Cross-instrumentation is the process of generating SystemTap instrumentation modules from a SystemTap script on one system to be used on another system. This process offers the following benefits:

  • The kernel information packages for various machines can be installed on a single host machine.
Caution

Kernel packaging bugs may prevent this. In such cases, the kernel-debuginfo and kernel-devel packages for the host system and target system must match. If this occurs, report the bug at https://bugzilla.redhat.com/.

  • Each target machine needs only one package to be installed to use the generated SystemTap instrumentation module: systemtap-runtime.
Important

The host system must be the same architecture and running the same distribution of Linux as the target system in order for the built instrumentation module to work.

Terminology
instrumentation module
The kernel module built from a SystemTap script; the SystemTap module is built on the host system, and will be loaded on the target kernel of the target system.
host system
The system on which the instrumentation modules (from SystemTap scripts) are compiled, to be loaded on target systems.
target system
The system in which the instrumentation module is being built (from SystemTap scripts).
target kernel
The kernel of the target system. This is the kernel that loads and runs the instrumentation module.

35.2. Initializing cross-instrumentation of SystemTap

Initialize cross-instrumentation of SystemTap to build SystemTap instrumentation modules from a SystemTap script on one system and use them on another system that does not have SystemTap fully deployed.

Prerequisites

  • SystemTap is installed on the host system as described in Installing Systemtap.
  • The systemtap-runtime package is installed on each target system:

    # yum install systemtap-runtime
  • Both the host system and target system are the same architecture.
  • Both the host system and target system are running the same major version of Red Hat Enterprise Linux (such as Red Hat Enterprise Linux 8), they can be running different minor versions (such as 8.1 and 8.2).
Caution

Kernel packaging bugs may prevent multiple kernel-debuginfo and kernel-devel packages from being installed on one system. In such cases, the minor version for the host system and target system must match. If this occurs, report the bug at https://bugzilla.redhat.com/.

Procedure

  1. Determine the kernel running on each target system:

    $ uname -r

    Repeat this step for each target system.

  2. On the host system, install the target kernel and related packages for each target system by the method described in Installing Systemtap.
  3. Build an instrumentation module on the host system, copy this module to and run this module on on the target system either:

    1. Using remote implementation:

      # stap --remote target_system script

      This command remotely implements the specified script on the target system. You must ensure an SSH connection can be made to the target system from the host system for this to be successful.

    2. Manually:

      1. Build the instrumentation module on the host system:

        # stap -r kernel_version script -m module_name -p 4

        Here, kernel_version refers to the version of the target kernel determined in step 1, script refers to the script to be converted into an instrumentation module, and module_name is the desired name of the instrumentation module. The -p4 option tells SystemTap to not load and run the compiled module.

      2. Once the instrumentation module is compiled, copy it to the target system and load it using the following command:

        # staprun module_name.ko