Chapter 9. Setting up graphical representation of PCP metrics

Using a combination of pcp, grafana, pcp redis, pcp bpftrace, and pcp vector provides provides graphical representation of the live data or data collected by Performance Co-Pilot (PCP).

9.1. Setting up PCP with pcp-zeroconf

This procedure describes how to set up PCP on a system with the pcp-zeroconf package. Once the pcp-zeroconf package is installed, the system records the default set of metrics into archived files.

Procedure

  • Install the pcp-zeroconf package: 

    # dnf install pcp-zeroconf

Verification steps

  • Ensure that the pmlogger service is active, and starts archiving the metrics: 

    # pcp | grep pmlogger
 pmlogger: primary logger: /var/log/pcp/pmlogger/localhost.localdomain/20200401.00.12

Additional resources

9.2. Setting up a grafana-server

Grafana generates graphs that are accessible from a browser. The grafana-server is a back-end server for the Grafana dashboard. It listens, by default, on all interfaces, and provides web services accessed through the web browser. The grafana-pcp plugin interacts with the pmproxy protocol in the backend.

This procedure describes how to set up a grafana-server.

Prerequisites

Procedure

  1. Install the following packages: 

    # dnf install grafana grafana-pcp

  2. Restart and enable the following service: 

    # systemctl restart grafana-server
# systemctl enable grafana-server

  3. Open the server’s firewall for network traffic to the Grafana service. 

    # firewall-cmd --permanent --add-service=grafana
success

# firewall-cmd --reload
success

Verification steps

  • Ensure that the grafana-server is listening and responding to requests: 

    # ss -ntlp | grep 3000
LISTEN  0  128  *:3000  *:*  users:(("grafana-server",pid=19522,fd=7))

  • Ensure that the grafana-pcp plugin is installed: 

    # grafana-cli plugins ls | grep performancecopilot-pcp-app

performancecopilot-pcp-app @ 3.1.0

Additional resources

  • pmproxy(1) and grafana-server man pages

9.3. Accessing the Grafana web UI

This procedure describes how to access the Grafana web interface.

Using the Grafana web interface, you can:

  • add PCP Redis, PCP bpftrace, and PCP Vector data sources
  • create dashboard
  • view an overview of any useful metrics
  • create alerts in PCP Redis

Prerequisites

  1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
  2. The grafana-server is configured. For more information, see Setting up a grafana-server.

Procedure

  1. On the client system, open a browser and access the grafana-server on port 3000, using http://192.0.2.0:3000 link.

    Replace 192.0.2.0 with your machine IP.

  2. For the first login, enter admin in both the Email or username and Password field.

    Grafana prompts to set a New password to create a secured account. If you want to set it later, click Skip.

  3. From the menu, hover over the    grafana gear icon    Configuration icon and then click Plugins.
  4. In the Plugins tab, type performance co-pilot in the Search by name or type text box and then click Performance Co-Pilot (PCP) plugin.
  5. In the Plugins / Performance Co-Pilot pane, click Enable.

  6. Click Grafana    grafana home page whirl icon    icon. The Grafana Home page is displayed.

    Figure 9.1. Home Dashboard

    grafana home dashboard
    Note

    The top corner of the screen has a similar    grafana top corner settings icon    icon, but it controls the general Dashboard settings.

  7. In the Grafana Home page, click Add your first data source to add PCP Redis, PCP bpftrace, and PCP Vector data sources. For more information about adding data source, see:

  8. Optional: From the menu, hover over the admin profile    grafana logout option icon    icon to change the Preferences including Edit Profile, Change Password, or to Sign out.

Additional resources

  • grafana-cli and grafana-server man pages

9.4. Configuring secure connections for Grafana

You can establish secure connections between Grafana and Performance Co-Pilot (PCP) components. Establishing secure connections between these components helps prevent unauthorized parties from accessing or modifying the data being collected and monitored.

Prerequisites

  • PCP is installed. For more information, see Installing and enabling PCP.
  • The grafana-server is configured. For more information, see Setting up a grafana-server.

  • The private client key is stored in the /etc/grafana/grafana.key file. If you use a different path, modify the path in the corresponding steps of the procedure.

    For details about creating a private key and certificate signing request (CSR), as well as how to request a certificate from a certificate authority (CA), see your CA’s documentation.

  • The TLS client certificate is stored in the /etc/grafana/grafana.crt file. If you use a different path, modify the path in the corresponding steps of the procedure.

Procedure

  1. As a root user, open the /etc/grafana/grana.ini file and adjust the following options in the [server] section to reflect the following: 

    protocol = https
cert_key = /etc/grafana/grafana.key
cert_file = /etc/grafana/grafana.crt

  2. Ensure grafana can access the certificates: 

    # su grafana -s /bin/bash -c \
  'ls -1 /etc/grafana/grafana.crt /etc/grafana/grafana.key'
/etc/grafana/grafana.crt
/etc/grafana/grafana.key

  3. Restart and enable the Grafana service to apply the configuration changes: 

    # systemctl restart grafana-server
# systemctl enable grafana-server

Verification

  1. On the client system, open a browser and access the grafana-server machine on port 3000, using the https://192.0.2.0:3000 link. Replace 192.0.2.0 with your machine IP.

  2. Confirm the    lock icon    lock icon is displayed beside the address bar.

    Note

    If the protocol is set to http and an HTTPS connection is attempted, you will receive a ERR_SSL_PROTOCOL_ERROR error. If the protocol is set to https and an HTTP connection is attempted, the Grafana server responds with a “Client sent an HTTP request to an HTTPS server” message.

9.5. Configuring PCP Redis

Use the PCP Redis data source to:

  • View data archives
  • Query time series using pmseries language
  • Analyze data across multiple hosts

Prerequisites

  1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
  2. The grafana-server is configured. For more information, see Setting up a grafana-server.
  3. Mail transfer agent, for example, sendmail or postfix is installed and configured.

Procedure

  1. Install the redis package: 

    # dnf install redis

  2. Start and enable the following services: 

    # systemctl start pmproxy redis
# systemctl enable pmproxy redis

  3. Restart the grafana-server: 

    # systemctl restart grafana-server

Verification steps

  • Ensure that the pmproxy and redis are working: 

    # pmseries disk.dev.read
2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df

    This command does not return any data if the redis package is not installed.

Additional resources

  • pmseries(1) man page

9.6. Configuring secure connections for PCP redis

You can establish secure connections between performance co-pilot (PCP), Grafana, and PCP redis. Establishing secure connections between these components helps prevent unauthorized parties from accessing or modifying the data being collected and monitored.

Prerequisites

  • PCP is installed. For more information, see Installing and enabling PCP.
  • The grafana-server is configured. For more information, see Setting up a grafana-server.
  • PCP redis is installed. For more information, see Configuring PCP Redis.

  • The private client key is stored in the /etc/redis/client.key file. If you use a different path, modify the path in the corresponding steps of the procedure.

    For details about creating a private key and certificate signing request (CSR), as well as how to request a certificate from a certificate authority (CA), see your CA’s documentation.

  • The TLS client certificate is stored in the /etc/redis/client.crt file. If you use a different path, modify the path in the corresponding steps of the procedure.
  • The TLS server key is stored in the /etc/redis/redis.key file. If you use a different path, modify the path in the corresponding steps of the procedure.
  • The TLS server certificate is stored in the /etc/redis/redis.crt file. If you use a different path, modify the path in the corresponding steps of the procedure.
  • The CA certificate is stored in the /etc/redis/ca.crt file. If you use a different path, modify the path in the corresponding steps of the procedure.

Additionally, for the pmproxy daemon:

  • The private server key is stored in the /etc/pcp/tls/server.key file. If you use a different path, modify the path in the corresponding steps of the procedure.

Procedure

  1. As a root user, open the /etc/redis/redis.conf file and adjust the TLS/SSL options to reflect the following properties: 

    port 0
tls-port 6379
tls-cert-file /etc/redis/redis.crt
tls-key-file /etc/redis/redis.key
tls-client-key-file /etc/redis/client.key
tls-client-cert-file /etc/redis/client.crt
tls-ca-cert-file /etc/redis/ca.crt

  2. Ensure redis can access the TLS certificates: 

    # su redis -s /bin/bash -c \
  'ls -1 /etc/redis/ca.crt /etc/redis/redis.key /etc/redis/redis.crt'
/etc/redis/ca.crt
/etc/redis/redis.crt
/etc/redis/redis.key

  3. Restart the redis server to apply the configuration changes: 

    # systemctl restart redis

Verification

  • Confirm the TLS configuration works: 

    # redis-cli --tls --cert /etc/redis/client.crt \
    --key /etc/redis/client.key \
    --cacert /etc/redis/ca.crt <<< "PING"
PONG

    Unsuccessful TLS configuration might result in the following error message: 

    Could not negotiate a TLS connection: Invalid CA Certificate File/Directory

9.7. Creating panels and alert in PCP Redis data source

After adding the PCP Redis data source, you can view the dashboard with an overview of useful metrics, add a query to visualize the load graph, and create alerts that help you to view the system issues after they occur.

Prerequisites

  1. The PCP Redis is configured. For more information, see Configuring PCP Redis.
  2. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.

Procedure

  1. Log into the Grafana web UI.
  2. In the Grafana Home page, click Add your first data source.
  3. In the Add data source pane, type redis in the Filter by name or type text box and then click PCP Redis.

  4. In the Data Sources / PCP Redis pane, perform the following:

    1. Add http://localhost:44322 in the URL field and then click Save & Test.

    2. Click Dashboards tab → Import → PCP Redis: Host Overview to see a dashboard with an overview of any useful metrics.

      Figure 9.2. PCP Redis: Host Overview

      pcp redis host overview

  5. Add a new panel:

    1. From the menu, hover over the    grafana plus sign    Create icon → Dashboard → Add new panel icon to add a panel.
    2. In the Query tab, select the PCP Redis from the query list instead of the selected default option and in the text field of A, enter metric, for example, kernel.all.load to visualize the kernel load graph.
    3. Optional: Add Panel title and Description, and update other options from the Settings.
    4. Click Save to apply changes and save the dashboard. Add Dashboard name.

    5. Click Apply to apply changes and go back to the dashboard.

      Figure 9.3. PCP Redis query panel

      pcp redis query panel

  6. Create an alert rule:

    1. In the PCP Redis query panel, click    redis alert icon    Alert and then click Create Alert.
    2. Edit the Name, Evaluate query, and For fields from the Rule, and specify the Conditions for your alert.

    3. Click Save to apply changes and save the dashboard. Click Apply to apply changes and go back to the dashboard.

      Figure 9.4. Creating alerts in the PCP Redis panel

      pcp redis query alert panel
    4. Optional: In the same panel, scroll down and click Delete icon to delete the created rule.

    5. Optional: From the menu, click    alerting bell icon    Alerting icon to view the created alert rules with different alert statuses, to edit the alert rule, or to pause the existing rule from the Alert Rules tab.

      To add a notification channel for the created alert rule to receive an alert notification from Grafana, see Adding notification channels for alerts.

9.8. Adding notification channels for alerts

By adding notification channels, you can receive an alert notification from Grafana whenever the alert rule conditions are met and the system needs further monitoring.

You can receive these alerts after selecting any one type from the supported list of notifiers, which includes DingDing, Discord, Email, Google Hangouts Chat, HipChat, Kafka REST Proxy, LINE, Microsoft Teams, OpsGenie, PagerDuty, Prometheus Alertmanager, Pushover, Sensu, Slack, Telegram, Threema Gateway, VictorOps, and webhook.

Prerequisites

  1. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.
  2. An alert rule is created. For more information, see Creating panels and alert in PCP Redis data source.

  3. Configure SMTP and add a valid sender’s email address in the grafana/grafana.ini file: 

    # vi /etc/grafana/grafana.ini

[smtp]
enabled = true
from_address = abc@gmail.com

    Replace abc@gmail.com by a valid email address.

  4. Restart grafana-server 

    # systemctl restart grafana-server.service

Procedure

  1. From the menu, hover over the    alerting bell icon    Alerting icon → click Contact Points → New contact point.

  2. In the New contact point details view, perform the following:

    1. Enter your name in the Name text box
    2. Select the Contact point type, for example, Email and enter the email address. You can add many email addresses by using the ; separator.
    3. Optional: Configure Optional Email settings and Notification settings.
  3. Click Save contact point.

  4. Select a notification channel in the alert rule:

    1. From the menu, select Notification policies icon and then click + New specific policy.
    2. Choose the Contact point you have just created
    3. Click the Save policy button

Additional resources

9.9. Setting up authentication between PCP components

You can setup authentication using the scram-sha-256 authentication mechanism, which is supported by PCP through the Simple Authentication Security Layer (SASL) framework.

Procedure

  1. Install the sasl framework for the scram-sha-256 authentication mechanism: 

    # dnf install cyrus-sasl-scram cyrus-sasl-lib

  2. Specify the supported authentication mechanism and the user database path in the pmcd.conf file: 

    # vi /etc/sasl2/pmcd.conf

mech_list: scram-sha-256

sasldb_path: /etc/pcp/passwd.db

  3. Create a new user: 

    # useradd -r metrics

    Replace metrics by your user name.

  4. Add the created user in the user database: 

    # saslpasswd2 -a pmcd metrics

Password:
Again (for verification):

    To add the created user, you are required to enter the metrics account password.

  5. Set the permissions of the user database: 

    # chown root:pcp /etc/pcp/passwd.db
# chmod 640 /etc/pcp/passwd.db

  6. Restart the pmcd service: 

    # systemctl restart pmcd

Verification steps

  • Verify the sasl configuration: 

    # pminfo -f -h "pcp://127.0.0.1?username=metrics" disk.dev.read
Password:
disk.dev.read
inst [0 or "sda"] value 19540

Additional resources

9.10. Installing PCP bpftrace

Install the PCP bpftrace agent to introspect a system and to gather metrics from the kernel and user-space tracepoints.

The bpftrace agent uses bpftrace scripts to gather the metrics. The bpftrace scripts use the enhanced Berkeley Packet Filter (eBPF).

This procedure describes how to install a pcp bpftrace.

Prerequisites

  1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
  2. The grafana-server is configured. For more information, see Setting up a grafana-server.
  3. The scram-sha-256 authentication mechanism is configured. For more information, see Setting up authentication between PCP components.

Procedure

  1. Install the pcp-pmda-bpftrace package: 

    # dnf install pcp-pmda-bpftrace

  2. Edit the bpftrace.conf file and add the user that you have created in the {setting-up-authentication-between-pcp-components}: 

    # vi /var/lib/pcp/pmdas/bpftrace/bpftrace.conf

[dynamic_scripts]
enabled = true
auth_enabled = true
allowed_users = root,metrics

    Replace metrics by your user name.

  3. Install bpftrace PMDA: 

    # cd /var/lib/pcp/pmdas/bpftrace/
# ./Install
Updating the Performance Metrics Name Space (PMNS) ...
Terminate PMDA if already installed ...
Updating the PMCD control file, and notifying PMCD ...
Check bpftrace metrics have appeared ... 7 metrics and 6 values

    The pmda-bpftrace is now installed, and can only be used after authenticating your user. For more information, see Viewing the PCP bpftrace System Analysis dashboard.

Additional resources

  • pmdabpftrace(1) and bpftrace man pages

9.11. Viewing the PCP bpftrace System Analysis dashboard

Using the PCP bpftrace data source, you can access the live data from sources which are not available as normal data from the pmlogger or archives

In the PCP bpftrace data source, you can view the dashboard with an overview of useful metrics.

Prerequisites

  1. The PCP bpftrace is installed. For more information, see Installing PCP bpftrace.
  2. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.

Procedure

  1. Log into the Grafana web UI.
  2. In the Grafana Home page, click Add your first data source.
  3. In the Add data source pane, type bpftrace in the Filter by name or type text box and then click PCP bpftrace.

  4. In the Data Sources / PCP bpftrace pane, perform the following:

    1. Add http://localhost:44322 in the URL field.
    2. Toggle the Basic Auth option and add the created user credentials in the User and Password field.

    3. Click Save & Test.

      Figure 9.5. Adding PCP bpftrace in the data source

      bpftrace auth

    4. Click Dashboards tab → Import → PCP bpftrace: System Analysis to see a dashboard with an overview of any useful metrics.

      Figure 9.6. PCP bpftrace: System Analysis

      pcp bpftrace bpftrace system analysis

9.12. Installing PCP Vector

This procedure describes how to install a pcp vector.

Prerequisites

  1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
  2. The grafana-server is configured. For more information, see Setting up a grafana-server.

Procedure

  1. Install the pcp-pmda-bcc package: 

    # dnf install pcp-pmda-bcc

  2. Install the bcc PMDA: 

    # cd /var/lib/pcp/pmdas/bcc
# ./Install
[Wed Apr  1 00:27:48] pmdabcc(22341) Info: Initializing, currently in 'notready' state.
[Wed Apr  1 00:27:48] pmdabcc(22341) Info: Enabled modules:
[Wed Apr  1 00:27:48] pmdabcc(22341) Info: ['biolatency', 'sysfork',
[...]
Updating the Performance Metrics Name Space (PMNS) ...
Terminate PMDA if already installed ...
Updating the PMCD control file, and notifying PMCD ...
Check bcc metrics have appeared ... 1 warnings, 1 metrics and 0 values

Additional resources

  • pmdabcc(1) man page

9.13. Viewing the PCP Vector Checklist

The PCP Vector data source displays live metrics and uses the pcp metrics. It analyzes data for individual hosts.

After adding the PCP Vector data source, you can view the dashboard with an overview of useful metrics and view the related troubleshooting or reference links in the checklist.

Prerequisites

  1. The PCP Vector is installed. For more information, see Installing PCP Vector.
  2. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.

Procedure

  1. Log into the Grafana web UI.
  2. In the Grafana Home page, click Add your first data source.
  3. In the Add data source pane, type vector in the Filter by name or type text box and then click PCP Vector.

  4. In the Data Sources / PCP Vector pane, perform the following:

    1. Add http://localhost:44322 in the URL field and then click Save & Test.

    2. Click Dashboards tab → Import → PCP Vector: Host Overview to see a dashboard with an overview of any useful metrics.

      Figure 9.7. PCP Vector: Host Overview

      pcp vector host overview

  5. From the menu, hover over the    pcp plugin in grafana    Performance Co-Pilot plugin and then click PCP Vector Checklist.

    In the PCP checklist, click    pcp vector checklist troubleshooting doc    help or    pcp vector checklist warning    warning icon to view the related troubleshooting or reference links.

    Figure 9.8. Performance Co-Pilot / PCP Vector Checklist

    pcp vector checklist

9.14. Troubleshooting Grafana issues

It is sometimes neccesary to troubleshoot Grafana issues, such as, Grafana does not display any data, the dashboard is black, or similar issues.

Procedure

  • Verify that the pmlogger service is up and running by executing the following command: 

    $ systemctl status pmlogger

  • Verify if files were created or modified to the disk by executing the following command: 

    $ ls /var/log/pcp/pmlogger/$(hostname)/ -rlt
total 4024
-rw-r--r--. 1 pcp pcp   45996 Oct 13  2019 20191013.20.07.meta.xz
-rw-r--r--. 1 pcp pcp     412 Oct 13  2019 20191013.20.07.index
-rw-r--r--. 1 pcp pcp   32188 Oct 13  2019 20191013.20.07.0.xz
-rw-r--r--. 1 pcp pcp   44756 Oct 13  2019 20191013.20.30-00.meta.xz
[..]

  • Verify that the pmproxy service is running by executing the following command: 

    $ systemctl status pmproxy

  • Verify that pmproxy is running, time series support is enabled, and a connection to Redis is established by viewing the /var/log/pcp/pmproxy/pmproxy.log file and ensure that it contains the following text: 

    pmproxy(1716) Info: Redis slots, command keys, schema version setup

    Here, 1716 is the PID of pmproxy, which will be different for every invocation of pmproxy.

  • Verify if the Redis database contains any keys by executing the following command: 

    $ redis-cli dbsize
(integer) 34837

  • Verify if any PCP metrics are in the Redis database and pmproxy is able to access them by executing the following commands: 

    $ pmseries disk.dev.read
2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df

$ pmseries "disk.dev.read[count:10]"
2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df
    [Mon Jul 26 12:21:10.085468000 2021] 117971 70e83e88d4e1857a3a31605c6d1333755f2dd17c
    [Mon Jul 26 12:21:00.087401000 2021] 117758 70e83e88d4e1857a3a31605c6d1333755f2dd17c
    [Mon Jul 26 12:20:50.085738000 2021] 116688 70e83e88d4e1857a3a31605c6d1333755f2dd17c
[...]
    $ redis-cli --scan --pattern "*$(pmseries 'disk.dev.read')"

pcp:metric.name:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df
pcp:values:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df
pcp:desc:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df
pcp:labelvalue:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df
pcp:instances:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df
pcp:labelflags:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df

  • Verify if there are any errors in the Grafana logs by executing the following command: 

    $ journalctl -e -u grafana-server
-- Logs begin at Mon 2021-07-26 11:55:10 IST, end at Mon 2021-07-26 12:30:15 IST. --
Jul 26 11:55:17 localhost.localdomain systemd[1]: Starting Grafana instance...
Jul 26 11:55:17 localhost.localdomain grafana-server[1171]: t=2021-07-26T11:55:17+0530 lvl=info msg="Starting Grafana" logger=server version=7.3.6 c>
Jul 26 11:55:17 localhost.localdomain grafana-server[1171]: t=2021-07-26T11:55:17+0530 lvl=info msg="Config loaded from" logger=settings file=/usr/s>
Jul 26 11:55:17 localhost.localdomain grafana-server[1171]: t=2021-07-26T11:55:17+0530 lvl=info msg="Config loaded from" logger=settings file=/etc/g>
[...]