Chapter 7. Monitoring performance with Performance Co-Pilot

Performance Co-Pilot (PCP) is a suite of tools, services, and libraries for monitoring, visualizing, storing, and analyzing system-level performance measurements.

As a system administrator, you can monitor the system’s performance using the PCP application in Red Hat Enterprise Linux 9.

7.1. Monitoring postfix with pmda-postfix

This procedure describes how to monitor performance metrics of the postfix mail server with pmda-postfix. It helps to check how many emails are received per second.

Prerequisites

Procedure

  1. Install the following packages:

    1. Install the pcp-system-tools: 

      # dnf install pcp-system-tools

    2. Install the pmda-postfix package to monitor postfix: 

      # dnf install pcp-pmda-postfix postfix

    3. Install the logging daemon: 

      # dnf install rsyslog

    4. Install the mail client for testing: 

      # dnf install mutt

  2. Enable the postfix and rsyslog services: 

    # systemctl enable postfix rsyslog
# systemctl restart postfix rsyslog

  3. Enable the SELinux boolean, so that pmda-postfix can access the required log files: 

    # setsebool -P pcp_read_generic_logs=on

  4. Install the PMDA: 

    # cd /var/lib/pcp/pmdas/postfix/

# ./Install

Updating the Performance Metrics Name Space (PMNS) ...
Terminate PMDA if already installed ...
Updating the PMCD control file, and notifying PMCD ...
Waiting for pmcd to terminate ...
Starting pmcd ...
Check postfix metrics have appeared ... 7 metrics and 58 values

Verification steps

  • Verify the pmda-postfix operation: 

    echo testmail | mutt root

  • Verify the available metrics: 

    # pminfo postfix

postfix.received
postfix.sent
postfix.queues.incoming
postfix.queues.maildrop
postfix.queues.hold
postfix.queues.deferred
postfix.queues.active

Additional resources

7.2. Visually tracing PCP log archives with the PCP Charts application

After recording metric data, you can replay the PCP log archives as graphs. The metrics are sourced from one or more live hosts with alternative options to use metric data from PCP log archives as a source of historical data. To customize the PCP Charts application interface to display the data from the performance metrics, you can use line plot, bar graphs, or utilization graphs.

Using the PCP Charts application, you can:

  • Replay the data in the PCP Charts application application and use graphs to visualize the retrospective data alongside live data of the system.
  • Plot performance metric values into graphs.
  • Display multiple charts simultaneously.

Prerequisites

Procedure

  1. Launch the PCP Charts application from the command line: 

    # pmchart

    Figure 7.1. PCP Charts application

    pmchart started

    The pmtime server settings are located at the bottom. The start and pause button allows you to control:

    • The interval in which PCP polls the metric data
    • The date and time for the metrics of historical data
  2. Click File and then New Chart to select metric from both the local machine and remote machines by specifying their host name or address. Advanced configuration options include the ability to manually set the axis values for the chart, and to manually choose the color of the plots.

  3. Record the views created in the PCP Charts application:

    Following are the options to take images or record the views created in the PCP Charts application:

    • Click File and then Export to save an image of the current view.
    • Click Record and then Start to start a recording. Click Record and then Stop to stop the recording. After stopping the recording, the recorded metrics are archived to be viewed later.

  4. Optional: In the PCP Charts application, the main configuration file, known as the view, allows the metadata associated with one or more charts to be saved. This metadata describes all chart aspects, including the metrics used and the chart columns. Save the custom view configuration by clicking File and then Save View, and load the view configuration later.

    The following example of the PCP Charts application view configuration file describes a stacking chart graph showing the total number of bytes read and written to the given XFS file system loop1: 

    #kmchart
version 1

chart title "Filesystem Throughput /loop1" style stacking antialiasing off
    plot legend "Read rate"   metric xfs.read_bytes   instance  "loop1"
    plot legend "Write rate"  metric xfs.write_bytes  instance  "loop1"

Additional resources

7.3. Collecting data from SQL server using PCP

The SQL Server agent is available in Performance Co-Pilot (PCP), which helps you to monitor and analyze database performance issues.

This procedure describes how to collect data for Microsoft SQL Server via pcp on your system.

Prerequisites

  • You have installed Microsoft SQL Server for Red Hat Enterprise Linux and established a 'trusted' connection to an SQL server.
  • You have installed the Microsoft ODBC driver for SQL Server for Red Hat Enterprise Linux.

Procedure

  1. Install PCP: 

    # dnf install pcp-zeroconf

  2. Install packages required for the pyodbc driver: 

    # dnf install python3-pyodbc

  3. Install the mssql agent:

    1. Install the Microsoft SQL Server domain agent for PCP: 

      # dnf install pcp-pmda-mssql

    2. Edit the /etc/pcp/mssql/mssql.conf file to configure the SQL server account’s username and password for the mssql agent. Ensure that the account you configure has access rights to performance data. 

      username: user_name
password: user_password

      Replace user_name with the SQL Server account and user_password with the SQL Server user password for this account.

  4. Install the agent: 

    # cd /var/lib/pcp/pmdas/mssql
# ./Install
Updating the Performance Metrics Name Space (PMNS) ...
Terminate PMDA if already installed ...
Updating the PMCD control file, and notifying PMCD ...
Check mssql metrics have appeared ... 168 metrics and 598 values
[...]

Verification steps

  • Using the pcp command, verify if the SQL Server PMDA (mssql) is loaded and running: 

    $ pcp
Performance Co-Pilot configuration on rhel.local:

platform: Linux rhel.local 4.18.0-167.el8.x86_64 #1 SMP Sun Dec 15 01:24:23 UTC 2019 x86_64
 hardware: 2 cpus, 1 disk, 1 node, 2770MB RAM
 timezone: PDT+7
 services: pmcd pmproxy
     pmcd: Version 5.0.2-1, 12 agents, 4 clients
     pmda: root pmcd proc pmproxy xfs linux nfsclient mmv kvm mssql
           jbd2 dm
 pmlogger: primary logger: /var/log/pcp/pmlogger/rhel.local/20200326.16.31
     pmie: primary engine: /var/log/pcp/pmie/rhel.local/pmie.log

  • View the complete list of metrics that PCP can collect from the SQL Server: 

    # pminfo mssql

  • After viewing the list of metrics, you can report the rate of transactions. For example, to report on the overall transaction count per second, over a five second time window: 

    # pmval -t 1 -T 5 mssql.databases.transactions
  • View the graphical chart of these metrics on your system by using the pmchart command. For more information, see Visually tracing PCP log archives with the PCP Charts application.

Additional resources

7.4. Generating PCP archives from sadc archives

You can use the sadf tool provided by the sysstat package to generate PCP archives from native sadc archives.

Prerequisites

  • A sadc archive has been created: 

    # /usr/lib64/sa/sadc 1 5 -

    In this example, sadc is sampling system data 1 time in a 5 second interval. The outfile is specified as - which results in sadc writing the data to the standard system activity daily data file. This file is named saDD and is located in the /var/log/sa directory by default.

Procedure

  • Generate a PCP archive from a sadc archive: 

    # sadf -l -O pcparchive=/tmp/recording -2

    In this example, using the -2 option results in sadf generating a PCP archive from a sadc archive recorded 2 days ago.

Verification steps

You can use PCP commands to inspect and analyze the PCP archive generated from a sadc archive as you would a native PCP archive. For example:

  • To show a list of metrics in the PCP archive generated from an sadc archive archive, run: 

    $ pminfo --archive /tmp/recording
Disk.dev.avactive
Disk.dev.read
Disk.dev.write
Disk.dev.blkread
[...]

  • To show the timespace of the archive and hostname of the PCP archive, run: 

    $ pmdumplog --label /tmp/recording
Log Label (Log Format Version 2)
Performance metrics from host shard
        commencing Tue Jul 20 00:10:30.642477 2021
        ending     Wed Jul 21 00:10:30.222176 2021

  • To plot performance metrics values into graphs, run: 

    $ pmchart --archive /tmp/recording