Chapter 7. Setting up graphical representation of PCP metrics
Using a combination of redis
, pcp
, bpftrace
, vector
and grafana
provides graphs, based on the live data or data collected by Performance Co-Pilot (PCP). It allows to access graphs of PCP metrics using a web browser.
- The PCP is a generic framework which collects, monitors, analyzes, and stores performance related metrics. For more information about the PCP and its components, see Monitoring performance with Performance Co-Pilot.
-
Redis is an
in-memory-database
. It is used to store data from the archived files which is easily accessible for the graph generation by the Grafana application. -
Bpftrace allows access to the live data from sources which are not available as normal data from the
pmlogger
or archives. - Vector provides access to the live data but does not provide access to data from past.
-
Grafana generates graphs that are accessible from a browser. The
grafana-server
is a component that listens, by default, on all interfaces, and provides web services accessed through the web browser. Thegrafana-pcp
plugin interacts with thepmproxy
protocol in the backend.
7.1. Setting up PCP on a system
This procedure describes how to set up PCP on a system with the pcp-zeroconf
package. Once the pcp-zeroconf
package is installed, the system records the default set of metrics into archived files.
Procedure
Install the
pcp-zeroconf
package:# yum install pcp-zeroconf
Verification steps
Ensure that the
pmlogger
service is active, and starts archiving the metrics:# pcp | grep pmlogger pmlogger: primary logger: /var/log/pcp/pmlogger/localhost.localdomain/20200401.00.12
Additional resources
-
The
pmlogger
man page. - For more information about the PCP and its components, see Monitoring performance with Performance Co-Pilot.
7.2. Setting up a grafana-server
This procedure describes how to set up a grafana-server
. It is a back-end server for the Grafana dashbaord. The grafana-server
component listens on the interface, and provides web services which are accessed via browser.
Prerequisites
-
The
pcp-zeroconf
package is installed. For more information, see Section 7.1, “Setting up PCP on a system”.
Procedure
Install the following packages:
# yum install redis grafana grafana-pcp
Restart and enable the following components:
# systemctl restart redis pmproxy grafana-server # systemctl enable redis pmproxy grafana-server
Verification steps
Ensure that the
grafana-server
is listening and responding to requests:# ss -ntlp | grep 3000 LISTEN 0 128 *:3000 *:* users:(("grafana-server",pid=19522,fd=7))
Additional resources
-
The
pmproxy
man page. -
The
grafana-server
man page.
7.3. Accessing the Grafana web UI
This procedure describes how to access the Grafana web interface and generate graphs using different types of data sources.
Prerequisites
-
The
pcp-zeroconf
package is installed. For more information, see Section 7.1, “Setting up PCP on a system”. -
The
grafana-server
is configured. For more information, see Section 7.2, “Setting up a grafana-server”.
Procedure
On the client system, open a browser and access the
grafana-server
on port3000
, using http://192.0.2.0:3000 link.Replace 192.0.2.0 with your machine IP.
For the first login, enter admin in both the username and password field.
Figure 7.1. Grafana login page
- To create a secured account, Grafana prompts to set a new password.
In the pane, hover on the
configuration icon > click Plugins > in the Filter by name or type, type performance co-pilot > click Performance Co-Pilot plugin and > click Enable to enable the
grafana-pcp
plugin.Figure 7.2. Home Dashboard
NoteThe top corner of the screen has a similar
icon, but it controls the general Dashboard settings.
-
Click the
icon to view the Home Dashboard.
In the Home Dashboard, click Add data source to add PCP Redis, PCP bpftrace, and PCP Vector data sources.
For more information on how to add PCP Redis, PCP bpftrace, and PCP Vector data sources, see:
-
Optional: In the pane, hover on the
icon to change the Preferences or to Sign out.
Verification steps
Ensure that the Performance Co-Pilot plugin is enabled:
# grafana-cli plugins ls | grep performancecopilot-pcp-app performancecopilot-pcp-app @ 1.0.5
Additional resources
-
The
grafana-cli
man page. -
The
grafana-server
man page.
7.4. Adding PCP Redis as a data source
The PCP Redis data source visualizes everything which the archive contains and queries time series capability provided by the pmseries
language. It analyzes data across multiple hosts. This procedure describes how to add PCP Redis as a data source and how to view the dashboard with an overview of useful metrics.
Prerequisites
-
The
grafana-server
is accessible. For more information, see Section 7.3, “Accessing the Grafana web UI”.
Procedure
Click the
icon > click Add data source > in the Filter by name or type, type redis > and click PCP Redis > in the URL field, accept the given suggestion http://localhost:44322 and > click Save & Test.
Figure 7.3. Adding PCP Redis in the data source
In the pane, hover on the
filter icon > click Manage > in the Filter Dashboard by name, type pcp redis > select PCP Redis Host Overview to see a dashboard with an overview of any useful metrics.
Figure 7.4. PCP Redis Host Overview
In the pane, hover on the
icon > click Dashboard option > click Add Query > from the Query list, select the PCP Redis and > in the text field of A, enter metric, for example,
kernel.all.load
to visualize the kernel load graph.Figure 7.5. PCP Redis query
Additional resources
-
The
pmseries
man page.
7.5. Setting up authentication between PCP components
PCP supports the scram-sha-256
authentication mechanism through the Simple Authentication Security Layer (SASL) framework. This procedure describes how to setup authentication using the scram-sha-256
authentication mechanism.
From Red Hat Enterprise Linux 8.3, PCP supports the scram-sha-256
authentication mechanism.
Prerequisites
Install the
sasl
framework for thescram-sha-256
authentication mechanism:# yum install cyrus-sasl-scram cyrus-sasl-lib
Procedure
Specify the supported authentication mechanism and the user database path in the
pmcd.conf
file:# vi /etc/sasl2/pmcd.conf mech_list: scram-sha-256 sasldb_path: /etc/pcp/passwd.db
Create a new user:
# useradd -r metrics
Replace metrics by your user name.
Add the created user in the user database:
# saslpasswd2 -a pmcd metrics Password: Again (for verification):
Set the permissions of the user database:
# chown root:pcp /etc/pcp/passwd.db # chmod 640 /etc/pcp/passwd.db
Restart the
pmcd
service:# systemctl restart pmcd
Verification steps
Verify the
sasl
configuration:# pminfo -f -h "pcp://127.0.0.1?username=metrics" disk.dev.read Password: disk.dev.read inst [0 or "sda"] value 19540
Additional resources
-
For more information, see the
saslauthd
,pminfo
, andsha256
man pages. - For more information on setting up authentication between PCP components in RHEL 8.2, see How can I setup authentication between PCP components, like PMDAs and pmcd? knowledgebase article.
7.6. Adding PCP bpftrace as a data source
The bpftrace
agent enables system introspection using the bpftrace
scripts, which uses the enhanced Berkeley Packet Filter (eBPF
) to gather metrics from the kernel and user-space tracepoints. This procedure describes how to add PCP bpftrace
as a data source and how to view the dashboard with an overview of any useful metrics.
Prerequisites
-
The
grafana-server
is accessible. For more information, see Section 7.3, “Accessing the Grafana web UI”. -
The
scram-sha-256
authentication mechanism is configured. For more information, see Section 7.5, “Setting up authentication between PCP components”.
Prerequisites
Install the
pcp-pmda-bpftrace
package:# yum install pcp-pmda-bpftrace
Procedure
Edit the
bpftrace.conf
file and add your user, which you have created in the Section 7.5, “Setting up authentication between PCP components”:# vi /var/lib/pcp/pmdas/bpftrace/bpftrace.conf [dynamic_scripts] enabled = true auth_enabled = true allowed_users = root,metrics
Replace metrics by your user name.
Install the
bpftrace
PMDA:# cd /var/lib/pcp/pmdas/bpftrace/ # ./Install Updating the Performance Metrics Name Space (PMNS) ... Terminate PMDA if already installed ... Updating the PMCD control file, and notifying PMCD ... Check bpftrace metrics have appeared ... 7 metrics and 6 values
The
pmda-bpftrace
is now installed, and can only be used after authenticating your user.- Log into the Grafana web UI. For more information, see Section 7.3, “Accessing the Grafana web UI”.
Click the
icon > click Add data source > in the Filter by name or type, type bpftrace > and click PCP bpftrace > in the URL field, accept the given suggestion http://localhost:44322.
Select the User and Password field and > click Save & Test.
option > add the created user credentials in theFigure 7.6. Adding PCP bpftrace in the data source
In the pane, hover on the
filter icon > click Manage > in the Filter Dashboard by name, type pcp bpftrace > select PCP bpftrace System Analysis to see a dashboard with an overview of useful metrics.
Figure 7.7. PCP bpftrace System Analysis
Additional resources
-
The
pmdabpftrace
man page. -
The
bpftrace
man page.
7.7. Adding PCP Vector as a data source
The PCP Vector data source displays live metrics and uses the pcp
metrics. It analyzes data for individual hosts. This procedure describes how to add a PCP Vector as a data source and how to view the dashboard with an overview of any useful metrics.
Prerequisites
-
The
grafana-server
is accessible. For more information, see Section 7.3, “Accessing the Grafana web UI”.
Procedure
Install the
pcp-pmda-bcc
package:# yum install pcp-pmda-bcc
Install the
bcc
PMDA:# cd /var/lib/pcp/pmdas/bcc # ./Install [Wed Apr 1 00:27:48] pmdabcc(22341) Info: Initializing, currently in 'notready' state. [Wed Apr 1 00:27:48] pmdabcc(22341) Info: Enabled modules: [Wed Apr 1 00:27:48] pmdabcc(22341) Info: ['biolatency', 'sysfork', [...] Updating the Performance Metrics Name Space (PMNS) ... Terminate PMDA if already installed ... Updating the PMCD control file, and notifying PMCD ... Check bcc metrics have appeared ... 1 warnings, 1 metrics and 0 values
- Log into the Grafana web UI. For more information, see Section 7.3, “Accessing the Grafana web UI”.
Click the
icon > click Add data source > in the Filter by name or type, type vector > and click PCP Vector > in the URL field, accept the given suggestion http://localhost:44322 and > click Save & Test.
Figure 7.8. Adding PCP Vector in the data source
In the pane, hover on the
icon > click Manage > in the Filter Dashboard by name, type pcp vector > select PCP Vector Host Overview to see a dashboard with an overview of useful metrics.
Figure 7.9. PCP Vector Host Overview
Additional resources
-
The
pmdabcc
man page.