Chapter 24. Performance

This chapter lists the most notable changes to performance between RHEL 8 and RHEL 9.

24.1. Notable changes to performance

Performance Co-Pilot rebased to version 6.0

Starting in RHEL 9.2, Performance Co-Pilot (PCP) has been updated to version 6.0. Notable improvements include:

  1. Version 3 PCP archive support:

    This includes support for instance domain change-deltas, Y2038-safe timestamps, nanosecond-precision timestamps, arbitrary timezones support, and 64-bit file offsets used throughout for larger (beyond 2GB) individual volumes.

    This feature is currently opt-in via the PCP_ARCHIVE_VERSION setting in the /etc/pcp.conf file.

    Version 2 archives remain the default.

  2. Only OpenSSL is used throughout PCP. Mozilla NSS/NSPR use has been dropped:

    This impacts libpcp, PMAPI clients and PMCD use of encryption. These elements are now configured and used consistently with pmproxy HTTPS support and redis-server, which were both already using OpenSSL.

  3. New nanosecond precision timestamp PMAPI calls for PCP library interfaces that make use of timestamps.

    These are all optional, and full backward compatibility is preserved for existing tools.

  4. The following tools and services have been updated:

    pcp2elasticsearch
    Implemented authentication support.
    pcp-dstat
    Implemented support for the top-alike plugins.
    pcp-htop
    Updated to the latest stable upstream release.
    pmseries
    Added sum, avg, stdev, nth_percentile, max_inst, max_sample, min_inst and min_sample functions.
    pmdabpf
    Added CO-RE (Compile Once - Run Everywhere) modules and support for AMD64, Intel 64-bit, 64-bit ARM, and IBM Power Systems.
    pmdabpftrace
    Moved example autostart scripts to the /usr/share directory.
    pmdadenki
    Added support for multiple active batteries.
    pmdalinux
    Updates for the latest /proc/net/netstat changes.
    pmdaopenvswitch
    Added additional interface and coverage statistics.
    pmproxy
    Request parameters can now be sent in the request body.
    pmieconf
    Added several pmie rules for Open vSwitch metrics.
    pmlogger_farm
    Added a default configuration file for farm loggers.
    pmlogger_daily_report
    Some major efficiency improvements.

The sadf(1) tool now generates PCP archives from native sadc(1) archives

Starting in RHEL 9, the sadf(1) tool provided by the sysstat package can generate PCP archives from native sadc(1) archives.

By default, when the - flag is used with sadc(1), sadc(1) writes the data to the standard system activity daily data file. This file is named saDD and is located in the /var/log/sa directory by default. Conversely, when no input datafile is specified, the sadf(1) tool uses the standard system activity daily data file to generate archives. Pass numbers as an argument to tell sadf(1) to generate archives from data recorded that specified number of days in the past.

  • To generate a PCP archive from an sadc(1) archive recorded 2 days ago, run: 

    # sadf -l -O pcparchive=/tmp/recording -2

  • To show a list of metrics in the PCP archive generated from an sadc(1) archive archive, run: 

    $ pminfo --archive /tmp/recording
Disk.dev.avactive
Disk.dev.read
Disk.dev.write
Disk.dev.blkread
[...]

  • To show the timespace of the archive and hostname of the PCP archive generated from an sadc(1) archive 

    $ pmdumplog --label /tmp/recording
Log Label (Log Format Version 2)
Performance metrics from host shard
        commencing Tue Jul 20 00:10:30.642477 2021
        ending     Wed Jul 21 00:10:30.222176 2021

  • You can then analyze the PCP archive generated from an sadc(1) archive by using PCP commands, for example: 

    $ pmchart --archive /tmp/recording

New PCP PMDA - pmdabpf

RHEL 9 is distributed with the pcp-pmda-bpf package, which provides the pmdabpf Performance Co-Pilot (PCP) Performance Metric Domain Agent (PMDA).

The pmdabpf PMDA extracts live performance data from eBPF programs utilizing BPF CO-RE (Compile Once - Run Everywhere), that is libbpf and BTF.