Chapter 5. Setting up graphical representation of PCP metrics

Using a combination of redis, pcp, bpftrace, vector and grafana provides graphs, based on the live data or data collected by Performance Co-Pilot (PCP). It allows to access graphs of PCP metrics using a web browser.

  • The PCP is a generic framework which collects, monitors, analyzes, and stores performance related metrics. For more information about the PCP and its components, see Monitoring performance with Performance Co-Pilot.
  • Redis is an in-memory-database. It is used to store data from the archived files which is easily accessible for the graph generation by the Grafana application.
  • Bpftrace allows access to the live data from sources which are not available as normal data from the pmlogger or archives.
  • Vector provides access to the live data but does not provide access to data from past.
  • Grafana generates graphs that are accessible from a browser. The grafana-server is a component that listens, by default, on all interfaces, and provides web services accessed through the web browser. The grafana-pcp plugin interacts with the pmproxy protocol in the backend.

5.1. Setting up PCP on a system

This procedure describes how to set up PCP on a system with the pcp-zeroconf package. Once the pcp-zeroconf package is installed, the system records the default set of metrics into archived files.

Procedure

  • Install the pcp-zeroconf package: 

    # yum install pcp-zeroconf

Verification steps

  • Ensure that the pmlogger service is active, and starts archiving the metrics: 

    # pcp | grep pmlogger
 pmlogger: primary logger: /var/log/pcp/pmlogger/localhost.localdomain/20200401.00.12

Additional resources

5.2. Setting up a grafana-server

This procedure describes how to set up a grafana-server. It is a back-end server for the Grafana dashbaord. The grafana-server component listens on the interface, and provides web services which are accessed via browser.

Prerequisites

Procedure

  1. Install the following packages: 

    # yum install redis grafana grafana-pcp

  2. Restart and enable the following components: 

    # systemctl restart redis pmproxy grafana-server
# systemctl enable redis pmproxy grafana-server

Verification steps

  • Ensure that the grafana-server is listening and responding to requests: 

    # ss -ntlp | grep 3000
LISTEN  0  128  *:3000  *:*  users:(("grafana-server",pid=19522,fd=7))

Additional resources

  • The pmproxy man page.
  • The grafana-server man page.

5.3. Accessing the Grafana web UI

This procedure describes how to access the Grafana web interface and generate graphs using different types of data sources.

Prerequisites

Procedure

  1. On the client system, open a browser and access the grafana-server on port 3000, using http://192.0.2.0:3000 link.

    Replace 192.0.2.0 with your machine IP.

  2. For the first login, enter admin in both the username and password field.

    Figure 5.1. Grafana login page

    grafana login
  3. To create a secured account, Grafana prompts to set a new password.

  4. In the pane, hover on the    grafana gear icon    configuration icon > click Plugins > in the Filter by name or type, type performance co-pilot > click Performance Co-Pilot plugin and > click Enable to enable the grafana-pcp plugin.

    Figure 5.2. Home Dashboard

    grafana dashboard
    Note

    The top corner of the screen has a similar    grafana top corner settings icon    icon, but it controls the general Dashboard settings.

  5. Click the    Grafana logo    icon to view the Home Dashboard.

  6. In the Home Dashboard, click Add data source to add PCP Redis, PCP bpftrace, and PCP Vector data sources.

    For more information on how to add PCP Redis, PCP bpftrace, and PCP Vector data sources, see:

  7. Optional: In the pane, hover on the    grafana logout option icon    icon to change the Preferences or to Sign out.

Verification steps

  • Ensure that the Performance Co-Pilot plugin is enabled: 

    # grafana-cli plugins ls | grep performancecopilot-pcp-app
performancecopilot-pcp-app @ 1.0.5

Additional resources

  • The grafana-cli man page.
  • The grafana-server man page.

5.4. Adding PCP Redis as a data source

The PCP Redis data source visualizes everything which the archive contains and queries time series capability provided by the pmseries language. It analyzes data across multiple hosts. This procedure describes how to add PCP Redis as a data source and how to view the dashboard with an overview of useful metrics.

Prerequisites

Procedure

  1. Click the    Grafana logo    icon > click Add data source > in the Filter by name or type, type redis > and click PCP Redis > in the URL field, accept the given suggestion http://localhost:44322 and > click Save & Test.

    Figure 5.3. Adding PCP Redis in the data source

    pcp redis data sources

  2. In the pane, hover on the    grafana 4 queries icon    filter icon > click Manage > in the Filter Dashboard by name, type pcp redis > select PCP Redis Host Overview to see a dashboard with an overview of any useful metrics.

    Figure 5.4. PCP Redis Host Overview

    PCP redis host overview

  3. In the pane, hover on the    plus sign    icon > click Dashboard option > click Add Query > from the Query list, select the PCP Redis and > in the text field of A, enter metric, for example, kernel.all.load to visualize the kernel load graph.

    Figure 5.5. PCP Redis query

    PCP redis query

Additional resources

  • The pmseries man page.

5.5. Adding PCP bpftrace as a data source

The bpftrace agent enables system introspection using the bpftrace scripts, which uses the enhanced Berkeley Packet Filter (eBPF) to gather metrics from kernel and user-space tracepoints. This procedure describes how to add PCP bpftrace as a data source and how to view the dashboard with an overview of any useful metrics.

Prerequisites

Procedure

  1. Install the pcp-pmda-bpftrace package: 

    # yum install pcp-pmda-bpftrace

  2. Edit the bpftrace.conf file and disable the authentication option: 

    # vi /var/lib/pcp/pmdas/bpftrace/bpftrace.conf

[authentication]
# NB. Disabling authentication is NOT RECOMMENDED, as everyone would be
# able to run bpftrace scripts AS ROOT!
enabled = false
    Note

    Do not disable the authentication option in production.

  3. Install the bpftrace PMDA: 

    # cd /var/lib/pcp/pmdas/bpftrace/
# ./Install
Updating the Performance Metrics Name Space (PMNS) ...
Terminate PMDA if already installed ...
Updating the PMCD control file, and notifying PMCD ...
Check bpftrace metrics have appeared ... 7 metrics and 6 values
  4. Log into the Grafana web UI. For more information, see Section 5.3, “Accessing the Grafana web UI”.

  5. Click the    Grafana logo    icon > click Add data source > in the Filter by name or type, type bpftrace > and click PCP bpftrace > in the URL field, accept the given suggestion http://localhost:44322 and > click Save & Test.

    Figure 5.6. Adding PCP bpftrace in the data source

    pcp bpftrace data source

  6. In the pane, hover on the    grafana 4 queries icon    filter icon > click Manage > in the Filter Dashboard by name, type pcp bpftrace > select PCP bpftrace System Analysis to see a dashboard with an overview of useful metrics.

    Figure 5.7. PCP bpftrace System Analysis

    pcp bpftrace bpftrace system analysis

Additional resources

  • The pmdabpftrace man page.
  • The bpftrace man page.

5.6. Adding PCP Vector as a data source

The PCP Vector data source displays live metrics and uses the pcp metrics. It analyzes data for individual hosts. This procedure describes how to add a PCP Vector as a data source and how to view the dashboard with an overview of any useful metrics.

Prerequisites

Procedure

  1. Install the pcp-pmda-bcc package: 

    # yum install pcp-pmda-bcc

  2. Install the bcc PMDA: 

    # cd /var/lib/pcp/pmdas/bcc
# ./Install
[Wed Apr  1 00:27:48] pmdabcc(22341) Info: Initializing, currently in 'notready' state.
[Wed Apr  1 00:27:48] pmdabcc(22341) Info: Enabled modules:
[Wed Apr  1 00:27:48] pmdabcc(22341) Info: ['biolatency', 'sysfork',
[...]
Updating the Performance Metrics Name Space (PMNS) ...
Terminate PMDA if already installed ...
Updating the PMCD control file, and notifying PMCD ...
Check bcc metrics have appeared ... 1 warnings, 1 metrics and 0 values
  3. Log into the Grafana web UI. For more information, see Section 5.3, “Accessing the Grafana web UI”.

  4. Click the    Grafana logo    icon > click Add data source > in the Filter by name or type, type vector > and click PCP Vector > in the URL field, accept the given suggestion http://localhost:44322 and > click Save & Test.

    Figure 5.8. Adding PCP Vector in the data source

    pcp vector

  5. In the pane, hover on the    grafana 4 queries icon    icon > click Manage > in the Filter Dashboard by name, type pcp vector > select PCP Vector Host Overview to see a dashboard with an overview of useful metrics.

    Figure 5.9. PCP Vector Host Overview

    pcp vector overview

Additional resources

  • The pmdabcc man page.