Chapter 7. Setting up graphical representation of PCP metrics

Using a combination of redis, pcp, bpftrace, vector and grafana provides graphs, based on the live data or data collected by Performance Co-Pilot (PCP). It allows to access graphs of PCP metrics using a web browser.

The PCP is a generic framework which collects, monitors, analyzes, and stores performance related metrics. For more information about the PCP and its components, see Monitoring performance with Performance Co-Pilot.
Redis is an in-memory-database. It is used to store data from the archived files which is easily accessible for the graph generation by the Grafana application.
Bpftrace allows access to the live data from sources which are not available as normal data from the pmlogger or archives.
Vector provides access to the live data but does not provide access to data from past.
Grafana generates graphs that are accessible from a browser. The grafana-server is a component that listens, by default, on all interfaces, and provides web services accessed through the web browser. The grafana-pcp plugin interacts with the pmproxy protocol in the backend.

7.1. Setting up PCP on a system

This procedure describes how to set up PCP on a system with the pcp-zeroconf package. Once the pcp-zeroconf package is installed, the system records the default set of metrics into archived files.

Procedure

Install the pcp-zeroconf package:
```
# yum install pcp-zeroconf
```

Verification steps

Ensure that the pmlogger service is active, and starts archiving the metrics:

# pcp | grep pmlogger
 pmlogger: primary logger: /var/log/pcp/pmlogger/localhost.localdomain/20200401.00.12

Additional resources

The pmlogger man page.
For more information about the PCP and its components, see Monitoring performance with Performance Co-Pilot.

7.2. Setting up a grafana-server

This procedure describes how to set up a grafana-server. It is a back-end server for the Grafana dashbaord. The grafana-server component listens on the interface, and provides web services which are accessed via browser.

Prerequisites

The pcp-zeroconf package is installed. For more information, see Section 7.1, “Setting up PCP on a system”.

Procedure

Install the following packages:
```
# yum install redis grafana grafana-pcp
```

Restart and enable the following components:

# systemctl restart redis pmproxy grafana-server
# systemctl enable redis pmproxy grafana-server

Verification steps

Ensure that the grafana-server is listening and responding to requests:

# ss -ntlp | grep 3000
LISTEN  0  128  *:3000  *:*  users:(("grafana-server",pid=19522,fd=7))

Additional resources

The pmproxy man page.
The grafana-server man page.

7.3. Accessing the Grafana web UI

This procedure describes how to access the Grafana web interface and generate graphs using different types of data sources.

Prerequisites

The pcp-zeroconf package is installed. For more information, see Section 7.1, “Setting up PCP on a system”.
The grafana-server is configured. For more information, see Section 7.2, “Setting up a grafana-server”.

Procedure

On the client system, open a browser and access the grafana-server on port 3000, using http://192.0.2.0:3000 link.
Replace 192.0.2.0 with your machine IP.
For the first login, enter admin in both the username and password field.
Figure 7.1. Grafana login page
To create a secured account, Grafana prompts to set a new password.
In the pane, hover on the configuration icon > click Plugins > in the Filter by name or type, type performance co-pilot > click Performance Co-Pilot plugin and > click Enable to enable the grafana-pcp plugin.
Figure 7.2. Home Dashboard
Note
The top corner of the screen has a similar icon, but it controls the general Dashboard settings.
Click the icon to view the Home Dashboard.
In the Home Dashboard, click Add data source to add PCP Redis, PCP bpftrace, and PCP Vector data sources.
For more information on how to add PCP Redis, PCP bpftrace, and PCP Vector data sources, see:
Optional: In the pane, hover on the icon to change the Preferences or to Sign out.

Verification steps

Ensure that the Performance Co-Pilot plugin is enabled:

# grafana-cli plugins ls | grep performancecopilot-pcp-app
performancecopilot-pcp-app @ 1.0.5

Additional resources

The grafana-cli man page.
The grafana-server man page.

7.4. Adding PCP Redis as a data source

The PCP Redis data source visualizes everything which the archive contains and queries time series capability provided by the pmseries language. It analyzes data across multiple hosts. This procedure describes how to add PCP Redis as a data source and how to view the dashboard with an overview of useful metrics.

Prerequisites

The grafana-server is accessible. For more information, see Section 7.3, “Accessing the Grafana web UI”.

Procedure

Click the icon > click Add data source > in the Filter by name or type, type redis > and click PCP Redis > in the URL field, accept the given suggestion http://localhost:44322 and > click Save & Test.
Figure 7.3. Adding PCP Redis in the data source
In the pane, hover on the filter icon > click Manage > in the Filter Dashboard by name, type pcp redis > select PCP Redis Host Overview to see a dashboard with an overview of any useful metrics.
Figure 7.4. PCP Redis Host Overview
In the pane, hover on the icon > click Dashboard option > click Add Query > from the Query list, select the PCP Redis and > in the text field of A, enter metric, for example, kernel.all.load to visualize the kernel load graph.
Figure 7.5. PCP Redis query

Additional resources

The pmseries man page.

7.5. Setting up authentication between PCP components

PCP supports the scram-sha-256 authentication mechanism through the Simple Authentication Security Layer (SASL) framework. This procedure describes how to setup authentication using the scram-sha-256 authentication mechanism.

Note

From Red Hat Enterprise Linux 8.3, PCP supports the scram-sha-256 authentication mechanism.

Prerequisites

Install the sasl framework for the scram-sha-256 authentication mechanism:
```
# yum install cyrus-sasl-scram cyrus-sasl-lib
```

Procedure

Specify the supported authentication mechanism and the user database path in the pmcd.conf file:
```
# vi /etc/sasl2/pmcd.conf

mech_list: scram-sha-256

sasldb_path: /etc/pcp/passwd.db
```
Create a new user:
```
# useradd -r metrics
```
Replace metrics by your user name.

Add the created user in the user database:

# saslpasswd2 -a pmcd metrics

Password:
Again (for verification):

Set the permissions of the user database:

# chown root:pcp /etc/pcp/passwd.db
# chmod 640 /etc/pcp/passwd.db

Restart the pmcd service:
```
# systemctl restart pmcd
```

Verification steps

Verify the sasl configuration:

# pminfo -f -h "pcp://127.0.0.1?username=metrics" disk.dev.read
Password:
disk.dev.read
inst [0 or "sda"] value 19540

Additional resources

For more information, see the saslauthd, pminfo, and sha256 man pages.
For more information on setting up authentication between PCP components in RHEL 8.2, see How can I setup authentication between PCP components, like PMDAs and pmcd? knowledgebase article.

7.6. Adding PCP bpftrace as a data source

The bpftrace agent enables system introspection using the bpftrace scripts, which uses the enhanced Berkeley Packet Filter (eBPF) to gather metrics from the kernel and user-space tracepoints. This procedure describes how to add PCP bpftrace as a data source and how to view the dashboard with an overview of any useful metrics.

Prerequisites

The grafana-server is accessible. For more information, see Section 7.3, “Accessing the Grafana web UI”.
The scram-sha-256 authentication mechanism is configured. For more information, see Section 7.5, “Setting up authentication between PCP components”.

Prerequisites

Install the pcp-pmda-bpftrace package:
```
# yum install pcp-pmda-bpftrace
```

Procedure

Edit the bpftrace.conf file and add your user, which you have created in the Section 7.5, “Setting up authentication between PCP components”:
```
# vi /var/lib/pcp/pmdas/bpftrace/bpftrace.conf

[dynamic_scripts]
enabled = true
auth_enabled = true
allowed_users = root,metrics
```
Replace metrics by your user name.

Install the bpftrace PMDA:

# cd /var/lib/pcp/pmdas/bpftrace/
# ./Install
Updating the Performance Metrics Name Space (PMNS) ...
Terminate PMDA if already installed ...
Updating the PMCD control file, and notifying PMCD ...
Check bpftrace metrics have appeared ... 7 metrics and 6 values

The pmda-bpftrace is now installed, and can only be used after authenticating your user.

Log into the Grafana web UI. For more information, see Section 7.3, “Accessing the Grafana web UI”.
Click the icon > click Add data source > in the Filter by name or type, type bpftrace > and click PCP bpftrace > in the URL field, accept the given suggestion http://localhost:44322.
Select the Basic Auth option > add the created user credentials in the User and Password field and > click Save & Test.
Figure 7.6. Adding PCP bpftrace in the data source
In the pane, hover on the filter icon > click Manage > in the Filter Dashboard by name, type pcp bpftrace > select PCP bpftrace System Analysis to see a dashboard with an overview of useful metrics.
Figure 7.7. PCP bpftrace System Analysis

Additional resources

The pmdabpftrace man page.
The bpftrace man page.

7.7. Adding PCP Vector as a data source

The PCP Vector data source displays live metrics and uses the pcp metrics. It analyzes data for individual hosts. This procedure describes how to add a PCP Vector as a data source and how to view the dashboard with an overview of any useful metrics.

Prerequisites

The grafana-server is accessible. For more information, see Section 7.3, “Accessing the Grafana web UI”.

Procedure

Install the pcp-pmda-bcc package:
```
# yum install pcp-pmda-bcc
```

Install the bcc PMDA:

# cd /var/lib/pcp/pmdas/bcc
# ./Install
[Wed Apr  1 00:27:48] pmdabcc(22341) Info: Initializing, currently in 'notready' state.
[Wed Apr  1 00:27:48] pmdabcc(22341) Info: Enabled modules:
[Wed Apr  1 00:27:48] pmdabcc(22341) Info: ['biolatency', 'sysfork',
[...]
Updating the Performance Metrics Name Space (PMNS) ...
Terminate PMDA if already installed ...
Updating the PMCD control file, and notifying PMCD ...
Check bcc metrics have appeared ... 1 warnings, 1 metrics and 0 values

Log into the Grafana web UI. For more information, see Section 7.3, “Accessing the Grafana web UI”.
Click the icon > click Add data source > in the Filter by name or type, type vector > and click PCP Vector > in the URL field, accept the given suggestion http://localhost:44322 and > click Save & Test.
Figure 7.8. Adding PCP Vector in the data source
In the pane, hover on the icon > click Manage > in the Filter Dashboard by name, type pcp vector > select PCP Vector Host Overview to see a dashboard with an overview of useful metrics.
Figure 7.9. PCP Vector Host Overview

Additional resources

The pmdabcc man page.

Log in to Your Red Hat Account

Red Hat Account

Customer Portal

Select Your Language

Infrastructure and Management

Cloud Computing

Storage

Runtimes

Integration and Automation

Services

Tools

Red Hat Insights

Red Hat Product Security Center

Security Updates

Resources

Customer Portal Community

Customer Events

Stories

Chapter 7. Setting up graphical representation of PCP metrics

7.1. Setting up PCP on a system

7.2. Setting up a grafana-server

7.3. Accessing the Grafana web UI

7.4. Adding PCP Redis as a data source

7.5. Setting up authentication between PCP components

7.6. Adding PCP bpftrace as a data source

7.7. Adding PCP Vector as a data source