Chapter 24. Performance

This chapter lists the most notable changes to performance between RHEL 8 and RHEL 9.

24.1. Notable changes to performance

Performance Co-Pilot rebased to version 6.0

Starting in RHEL 9.2, Performance Co-Pilot (PCP) has been updated to version 6.0. Notable improvements include:

  1. Version 3 PCP archive support:

    This includes support for instance domain change-deltas, Y2038-safe timestamps, nanosecond-precision timestamps, arbitrary timezones support, and 64-bit file offsets used throughout for larger (beyond 2GB) individual volumes.

    This feature is currently opt-in via the PCP_ARCHIVE_VERSION setting in the /etc/pcp.conf file.

    Version 2 archives remain the default.

  2. Only OpenSSL is used throughout PCP. Mozilla NSS/NSPR use has been dropped:

    This impacts libpcp, PMAPI clients and PMCD use of encryption. These elements are now configured and used consistently with pmproxy HTTPS support and redis-server, which were both already using OpenSSL.

  3. New nanosecond precision timestamp PMAPI calls for PCP library interfaces that make use of timestamps.

    These are all optional, and full backward compatibility is preserved for existing tools.

  4. The following tools and services have been updated:

    Implemented authentication support.
    Implemented support for the top-alike plugins.
    Updated to the latest stable upstream release.
    Added sum, avg, stdev, nth_percentile, max_inst, max_sample, min_inst and min_sample functions.
    Added CO-RE (Compile Once - Run Everywhere) modules and support for AMD64, Intel 64-bit, 64-bit ARM, and IBM Power Systems.
    Moved example autostart scripts to the /usr/share directory.
    Added support for multiple active batteries.
    Updates for the latest /proc/net/netstat changes.
    Added additional interface and coverage statistics.
    Request parameters can now be sent in the request body.
    Added several pmie rules for Open vSwitch metrics.
    Added a default configuration file for farm loggers.
    Some major efficiency improvements.

The sadf(1) tool now generates PCP archives from native sadc(1) archives

Starting in RHEL 9, the sadf(1) tool provided by the sysstat package can generate PCP archives from native sadc(1) archives.

By default, when the - flag is used with sadc(1), sadc(1) writes the data to the standard system activity daily data file. This file is named saDD and is located in the /var/log/sa directory by default. Conversely, when no input datafile is specified, the sadf(1) tool uses the standard system activity daily data file to generate archives. Pass numbers as an argument to tell sadf(1) to generate archives from data recorded that specified number of days in the past.

  • To generate a PCP archive from an sadc(1) archive recorded 2 days ago, run:

    # sadf -l -O pcparchive=/tmp/recording -2
  • To show a list of metrics in the PCP archive generated from an sadc(1) archive archive, run:

    $ pminfo --archive /tmp/recording
  • To show the timespace of the archive and hostname of the PCP archive generated from an sadc(1) archive

    $ pmdumplog --label /tmp/recording
    Log Label (Log Format Version 2)
    Performance metrics from host shard
            commencing Tue Jul 20 00:10:30.642477 2021
            ending     Wed Jul 21 00:10:30.222176 2021
  • You can then analyze the PCP archive generated from an sadc(1) archive by using PCP commands, for example:

    $ pmchart --archive /tmp/recording

New PCP PMDA - pmdabpf

RHEL 9 is distributed with the pcp-pmda-bpf package, which provides the pmdabpf Performance Co-Pilot (PCP) Performance Metric Domain Agent (PMDA).

The pmdabpf PMDA extracts live performance data from eBPF programs utilizing BPF CO-RE (Compile Once - Run Everywhere), that is libbpf and BTF.