Chapter 7. Setting up graphical representation of PCP metrics

Using a combination of redis, pcp, bpftrace, vector and grafana provides graphs, based on the live data or data collected by Performance Co-Pilot (PCP). It allows to access graphs of PCP metrics using a web browser.

  • The PCP is a generic framework which collects, monitors, analyzes, and stores performance related metrics. For more information about the PCP and its components, see Monitoring performance with Performance Co-Pilot.
  • Redis is an in-memory-database. It is used to store data from the archived files which is easily accessible for the graph generation by the Grafana application.
  • Bpftrace allows access to the live data from sources which are not available as normal data from the pmlogger or archives.
  • Vector provides access to the live data but does not provide access to data from past.
  • Grafana generates graphs that are accessible from a browser. The grafana-server is a component that listens, by default, on all interfaces, and provides web services accessed through the web browser. The grafana-pcp plugin interacts with the pmproxy protocol in the backend.

7.1. Setting up PCP on a system

This procedure describes how to set up PCP on a system with the pcp-zeroconf package. Once the pcp-zeroconf package is installed, the system records the default set of metrics into archived files.

Procedure

  • Install the pcp-zeroconf package:

    # yum install pcp-zeroconf

Verification steps

  • Ensure that the pmlogger service is active, and starts archiving the metrics:

    # pcp | grep pmlogger
     pmlogger: primary logger: /var/log/pcp/pmlogger/localhost.localdomain/20200401.00.12

Additional resources

7.2. Setting up a grafana-server

This procedure describes how to set up a grafana-server. It is a back-end server for the Grafana dashbaord. The grafana-server component listens on the interface, and provides web services which are accessed via browser.

Prerequisites

Procedure

  1. Install the following packages:

    # yum install grafana grafana-pcp
  2. Restart and enable the following service:

    # systemctl restart grafana-server
    # systemctl enable grafana-server

Verification steps

  • Ensure that the grafana-server is listening and responding to requests:

    # ss -ntlp | grep 3000
    LISTEN  0  128  *:3000  *:*  users:(("grafana-server",pid=19522,fd=7))
  • Ensure that the Performance Co-Pilot plugin is installed:

    # grafana-cli plugins ls | grep performancecopilot-pcp-app
    
    performancecopilot-pcp-app @ 1.0.5

Additional resources

  • The pmproxy man page.
  • The grafana-server man page.

7.3. Accessing the Grafana web UI

This procedure describes how to access the Grafana web interface and generate graphs using different types of data sources.

Prerequisites

Procedure

  1. On the client system, open a browser and access the grafana-server on port 3000, using http://192.0.2.0:3000 link.

    Replace 192.0.2.0 with your machine IP.

  2. For the first login, enter admin in both the username and password field.

    Figure 7.1. Grafana login page

    grafana login
  3. To create a secured account, Grafana prompts to set a new password.
  4. From the menu, hover over the    grafana gear icon    configuration icon > click Plugins > in the Filter by name or type, type performance co-pilot > click Performance Co-Pilot plugin and > click Enable to enable the grafana-pcp plugin.

    Figure 7.2. Home Dashboard

    grafana dashboard
    Note

    The top corner of the screen has a similar    grafana top corner settings icon    icon, but it controls the general Dashboard settings.

  5. Click the    Grafana logo    icon to view the Home Dashboard.
  6. In the Home Dashboard, click Add data source to add PCP Redis, PCP bpftrace, and PCP Vector data sources.

    For more information on how to add PCP Redis, PCP bpftrace, and PCP Vector data sources, see:

  7. Optional: In the pane, hover on the    grafana logout option icon    icon to change the Preferences or to Sign out.

Additional resources

  • The grafana-cli man page.
  • The grafana-server man page.

7.4. Adding PCP Redis as a data source

The PCP Redis data source visualizes everything which the archive contains and queries time series capability provided by the pmseries language. It analyzes data across multiple hosts. This procedure describes how to add PCP Redis as a data source and how to view the dashboard with an overview of useful metrics.

Prerequisites

Procedure

  1. Install the redis package:

    # yum install redis
  2. Start and enable the following services:

    # systemctl start pmproxy redis
    # systemctl enable pmproxy redis
  3. Log into the Grafana web UI. For more information, see Section 7.3, “Accessing the Grafana web UI”.
  4. Click the    Grafana logo    icon > click Add data source > in the Filter by name or type, type redis > and click PCP Redis > in the URL field, accept the given suggestion http://localhost:44322 and > click Save & Test.

    Figure 7.3. Adding PCP Redis in the data source

    pcp redis data sources
  5. In the pane, hover on the    grafana 4 queries icon    filter icon > click Manage > in the Filter Dashboard by name, type pcp redis > select PCP Redis Host Overview to see a dashboard with an overview of any useful metrics.

    Figure 7.4. PCP Redis Host Overview

    PCP redis host overview
  6. In the pane, hover on the    plus sign    icon > click Dashboard option > click Add Query > from the Query list, select the PCP Redis and > in the text field of A, enter metric, for example, kernel.all.load to visualize the kernel load graph.

    Figure 7.5. PCP Redis query

    PCP redis query

Additional resources

  • The pmseries man page.

7.5. Setting up authentication between PCP components

PCP supports the scram-sha-256 authentication mechanism through the Simple Authentication Security Layer (SASL) framework. This procedure describes how to setup authentication using the scram-sha-256 authentication mechanism.

Note

From Red Hat Enterprise Linux 8.3, PCP supports the scram-sha-256 authentication mechanism.

Prerequisites

  • Install the sasl framework for the scram-sha-256 authentication mechanism:

    # yum install cyrus-sasl-scram cyrus-sasl-lib

Procedure

  1. Specify the supported authentication mechanism and the user database path in the pmcd.conf file:

    # vi /etc/sasl2/pmcd.conf
    
    mech_list: scram-sha-256
    
    sasldb_path: /etc/pcp/passwd.db
  2. Create a new user:

    # useradd -r metrics

    Replace metrics by your user name.

  3. Add the created user in the user database:

    # saslpasswd2 -a pmcd metrics
    
    Password:
    Again (for verification):
  4. Set the permissions of the user database:

    # chown root:pcp /etc/pcp/passwd.db
    # chmod 640 /etc/pcp/passwd.db
  5. Restart the pmcd service:

    # systemctl restart pmcd

Verification steps

  • Verify the sasl configuration:

    # pminfo -f -h "pcp://127.0.0.1?username=metrics" disk.dev.read
    Password:
    disk.dev.read
    inst [0 or "sda"] value 19540

Additional resources

7.6. Adding PCP bpftrace as a data source

The bpftrace agent enables system introspection using the bpftrace scripts, which uses the enhanced Berkeley Packet Filter (eBPF) to gather metrics from the kernel and user-space tracepoints. This procedure describes how to add PCP bpftrace as a data source and how to view the dashboard with an overview of any useful metrics.

Prerequisites

Procedure

  1. Install the pcp-pmda-bpftrace package:

    # yum install pcp-pmda-bpftrace
  2. Edit the bpftrace.conf file and add your user, which you have created in the Section 7.5, “Setting up authentication between PCP components”:

    # vi /var/lib/pcp/pmdas/bpftrace/bpftrace.conf
    
    [dynamic_scripts]
    enabled = true
    auth_enabled = true
    allowed_users = root,metrics

    Replace metrics by your user name.

  3. Install the bpftrace PMDA:

    # cd /var/lib/pcp/pmdas/bpftrace/
    # ./Install
    Updating the Performance Metrics Name Space (PMNS) ...
    Terminate PMDA if already installed ...
    Updating the PMCD control file, and notifying PMCD ...
    Check bpftrace metrics have appeared ... 7 metrics and 6 values

    The pmda-bpftrace is now installed, and can only be used after authenticating your user.

  4. Log into the Grafana web UI. For more information, see Section 7.3, “Accessing the Grafana web UI”.
  5. Click the    Grafana logo    icon > click Add data source > in the Filter by name or type, type bpftrace > and click PCP bpftrace > in the URL field, accept the given suggestion http://localhost:44322.

    Select the Basic Auth option > add the created user credentials in the User and Password field and > click Save & Test.

    Figure 7.6. Adding PCP bpftrace in the data source

    bpftrace auth
  6. In the pane, hover on the    grafana 4 queries icon    filter icon > click Manage > in the Filter Dashboard by name, type pcp bpftrace > select PCP bpftrace System Analysis to see a dashboard with an overview of useful metrics.

    Figure 7.7. PCP bpftrace System Analysis

    pcp bpftrace bpftrace system analysis

Additional resources

  • The pmdabpftrace man page.
  • The bpftrace man page.

7.7. Adding PCP Vector as a data source

The PCP Vector data source displays live metrics and uses the pcp metrics. It analyzes data for individual hosts. This procedure describes how to add a PCP Vector as a data source and how to view the dashboard with an overview of any useful metrics.

Prerequisites

Procedure

  1. Install the pcp-pmda-bcc package:

    # yum install pcp-pmda-bcc
  2. Install the bcc PMDA:

    # cd /var/lib/pcp/pmdas/bcc
    # ./Install
    [Wed Apr  1 00:27:48] pmdabcc(22341) Info: Initializing, currently in 'notready' state.
    [Wed Apr  1 00:27:48] pmdabcc(22341) Info: Enabled modules:
    [Wed Apr  1 00:27:48] pmdabcc(22341) Info: ['biolatency', 'sysfork',
    [...]
    Updating the Performance Metrics Name Space (PMNS) ...
    Terminate PMDA if already installed ...
    Updating the PMCD control file, and notifying PMCD ...
    Check bcc metrics have appeared ... 1 warnings, 1 metrics and 0 values
  3. Log into the Grafana web UI. For more information, see Section 7.3, “Accessing the Grafana web UI”.
  4. Click the    Grafana logo    icon > click Add data source > in the Filter by name or type, type vector > and click PCP Vector > in the URL field, accept the given suggestion http://localhost:44322 and > click Save & Test.

    Figure 7.8. Adding PCP Vector in the data source

    pcp vector
  5. In the pane, hover on the    grafana 4 queries icon    icon > click Manage > in the Filter Dashboard by name, type pcp vector > select PCP Vector Host Overview to see a dashboard with an overview of useful metrics.

    Figure 7.9. PCP Vector Host Overview

    pcp vector overview

Additional resources

  • The pmdabcc man page.