Chapter 3. Installing PCP Packages

This procedure describes how to install the PCP packages.

Prerequisites

  • Ensure you have a minimum of 20 GB space available in the /var/log/pcp directory.

    The default PCP data retention policy is to retain only that data collected during the past 14 days. Data storage per day is estimated to use usually between 100 MB and 500 MB of disk space, but may use up to several gigabytes.

  • Ensure that the base system on which Satellite Server is running is Red Hat Enterprise Linux 7.6. or later. The minimum supported version for the PCP packages is PCP version 4.1.

Procedure

  1. Enable the Red Hat Enterprise Linux optional repository:

    # subscription-manager repos --enable rhel-7-server-optional-rpms
  2. Install the PCP packages:

    # satellite-maintain packages install pcp \
      pcp-pmda-apache \
      pcp-pmda-postgresql \
      pcp-system-tools \
      pcp-webjs
  3. Enable and start the Performance Metrics Collector daemon, and the Performance Metrics Logger daemon:

    # systemctl enable pmcd pmlogger
    # systemctl start pmcd pmlogger

3.1. Configuring PCP Data Collection

This procedure describes how to configure PCP to collect metrics about processes, Satellite, Apache HTTP Server, and PostgreSQL.

Procedure

  1. To configure PCP to collect data about Satellite processes, create the /var/lib/pcp/pmdas/proc/hotproc.conf file and include the following content:

    #pmdahotproc
    Version 1.0
    
    # processes with load 0.9 or 1GB RSS memory
    (cpuburn > 0.9 || residentsize > 1000000) &&
    (
      fname == "java" ||
      fname == "qdrouterd" ||
      fname == "qpidd" ||
      (fname == "postgres" && psargs ~ /-D/) ||
      fname == "mongod" ||
      fname == "dynflow_executor" ||
      fname == "dynflow_executor_monitor" ||
      fname ~ /smart_proxy_dynflow_core/ ||
      psargs ~ /Passenger RackApp/ ||
      psargs ~ /celery beat/ ||
      psargs ~ /celery worker/ ||
      psargs ~ /pulp/ ||
      psargs ~ /smart-proxy/ ||
      psargs ~ /squid.conf/
    )

    By default, PCP collects basic system metrics. This step enables detailed metrics about the following Satellite processes:

    • Java
    • PostgreSQL
    • MongoDB
    • Dynflow
    • Passenger
    • Pulp
    • Qpid
  2. Configure PCP to log the process metrics being collected.

    # mkdir -p /var/lib/pcp/config/pmlogconf/foreman-hotproc
    # cat >/var/lib/pcp/config/pmlogconf/foreman-hotproc/summary << EOF
    #pmlogconf-setup 2.0
    ident   foreman hotproc metrics
    probe   hotproc.control.config != "" ? include : exclude
            hotproc.psinfo.psargs
            hotproc.psinfo.cnswap
            hotproc.psinfo.nswap
            hotproc.psinfo.rss
            hotproc.psinfo.vsize
            hotproc.psinfo.cstime
            hotproc.psinfo.cutime
            hotproc.psinfo.stime
            hotproc.psinfo.utime
            hotproc.io.write_bytes
            hotproc.io.read_bytes
            hotproc.schedstat.cpu_time
            hotproc.fd.count
    EOF
  3. Install the process monitoring PMDA.

    # cd /var/lib/pcp/pmdas/proc
    # ./Install
  4. Configure PCP to collect metrics from Apache HTTP Server.

    1. Enable the Apache HTTP Server extended status module.

      #cat >/etc/httpd/conf.d/01-status.conf <<EOF
      ExtendedStatus On
      LoadModule status_module modules/mod_status.so
      
      <Location "/server-status">
      PassengerEnabled off
      SetHandler server-status
      Order deny,allow
      Deny from all
      Allow from localhost
      </Location>
      EOF
    2. Enable the Apache HTTP Server PMDA.

      # cd /var/lib/pcp/pmdas/apache
      # ./Install
    3. Prevent the Satellite installer overwriting the extended status module’s configuration file.

      Add the following line to the /etc/foreman-installer/custom-hiera.yaml configuration file.

      apache::purge_configs: false
  5. Configure PCP to collect metrics from PostgreSQL.

    1. Change to the /var/lib/pcp/pmdas/postgresql directory.

      # cd /var/lib/pcp/pmdas/postgresql
    2. Run the installer.

      # ./Install
    3. Configure the PCP database interface to permit access to the PostgreSQL database.

      Edit the /etc/pcpdbi.conf configuration file, inserting the following lines:

      $database = "dbi:Pg:dbname=foreman;host=localhost";
      $username = "foreman";
      $password = "6qXfN9m5nii5iEcbz8nuiJBNsyjjdRHA"; 1
      $os_user = "foreman";
      1
      The value for $password is stored in /etc/foreman/database.yml configuration file.
    4. Change the SELinux pcp_pmcd_t domain permission to permit PCP access to the PostgreSQL database.

      # semanage permissive -a pcp_pmcd_t
    5. Verify the PostgreSQL PMDA is able to connect to PostgreSQL.

      Examine the /var/log/pcp/pmcd/postgresql.log file to confirm the connection is established. Without a successful database connection, the PostgreSQL PMDA will remain active, but not be able to provide any metrics.

      [Tue Aug 14 09:21:06] pmdapostgresql(25056) Info: PostgreSQL connection established

      If you find errors in /var/log/pcp/pmcd/postgresql.log, restart the pmcd service.

      # systemctl restart pmcd
  6. Enable telemetry functionality in Satellite.

    To enable collection of metrics from Satellite, you must send metrics via the statsd protocol into the pcp-mmvstatsd daemon. The metrics are aggregated and available via the PCP MMV API.

    1. Install the Foreman Telemetry and pcp-mmvstatsd packages.

      # satellite-maintain packages install foreman-telemetry pcp-mmvstatsd
    2. Enable and start the pcp-mmvstatsd service.

      # systemctl enable pcp-mmvstatsd
      # systemctl start pcp-mmvstatsd
    3. Enable the Satellite telemetry functionality.

      Add the following lines to /etc/foreman/settings.yaml configuration file:

      :telemetry:
        :prefix: 'fm_rails'
        :statsd:
          :enabled: true
          :host: '127.0.0.1:8125'
          :protocol: 'statsd'
        :prometheus:
          :enabled: false
        :logger:
          :enabled: false
          :level: 'INFO'
  7. Schedule daily storage of metrics in archive files:

    # cat >/etc/cron.daily/refresh_mmv <<EOF
    #!/bin/bash
    echo "log mandatory on 1 minute mmv" | /usr/bin/pmlc -P
    EOF
    # chmod +x /etc/cron.daily/refresh_mmv
  8. Restart the Apache HTTP Server and PCP to begin data collection:

    # systemctl restart httpd pmcd pmlogger

3.2. Enabling Access to Metrics via the Web UI

This procedure describes how to access metrics collected by PCP, via the web UI.

Procedure

  1. Enable the Red Hat Enterprise Linux optional repository:

    # subscription-manager repos --enable rhel-7-server-optional-rpms
  2. Install the PCP web API and applications:

    # satellite-maintain packages install pcp-webapi pcp-webapp-grafana pcp-webapp-vector
  3. Start and enable the PCP web service:

    # systemctl start pmwebd
    # systemctl enable pmwebd
  4. Open firewall port to allow access to the PCP web service:

    # firewall-cmd --add-port=44323/tcp
    # firewall-cmd --permanent --add-port=44323/tcp

3.3. Verifying PCP Configuration

To verify PCP is configured correctly, and services are active, run the following command:

# pcp

This outputs a summary of the active PCP configuration.

Example output from the pcp command:

Performance Co-Pilot configuration on satellite.example.com:

 platform: Linux satellite.example.com 3.10.0-862.3.3.el7.x86_64 #1 SMP Wed Jun 13 05:44:23 EDT 2018 x86_64
 hardware: 8 cpus, 4 disks, 1 node, 23380MB RAM
 timezone: AEST-10
 services: pmcd pmwebd
     pmcd: Version 3.12.2-1, 9 agents, 1 client
     pmda: root pmcd proc xfs linux apache mmv postgresql jbd2
 pmlogger: primary logger: /var/log/pcp/pmlogger/satellite.example.com/20180802.00.10

In this example, both the Performance Metrics Collector Daemon (pmcd), and the Performance Metrics Web Daemon (pmwebd) services are running. It also confirms the PMDAs which are collecting metrics. Finally, it lists the currently actively archive file, in which pmlogger is storing metrics.