Red Hat Training

A Red Hat training course is available for RHEL 8

Chapter 5. Monitoring performance using RHEL System Roles

As a system administrator, you can use use the metrics RHEL System Role with any Ansible Automation Platform control node to monitor the performance of a system.

5.1. Introduction to RHEL System Roles

RHEL System Roles is a collection of Ansible roles and modules. RHEL System Roles provide a configuration interface to remotely manage multiple RHEL systems. The interface enables managing system configurations across multiple versions of RHEL, as well as adopting new major releases.

On Red Hat Enterprise Linux 8, the interface currently consists of the following roles:

  • kdump
  • network
  • selinux
  • storage
  • certificate
  • kernel_settings
  • logging
  • metrics
  • nbde_client and nbde_server
  • timesync
  • tlog

All these roles are provided by the rhel-system-roles package available in the AppStream repository.

Additional resources

5.2. RHEL System Roles terminology

You can find the following terms across this documentation:

System Roles terminology

Ansible playbook
Playbooks are Ansible’s configuration, deployment, and orchestration language. They can describe a policy you want your remote systems to enforce, or a set of steps in a general IT process.
Control node
Any machine with Ansible installed. You can run commands and playbooks, invoking /usr/bin/ansible or /usr/bin/ansible-playbook, from any control node. You can use any computer that has Python installed on it as a control node - laptops, shared desktops, and servers can all run Ansible. However, you cannot use a Windows machine as a control node. You can have multiple control nodes.
Inventory
A list of managed nodes. An inventory file is also sometimes called a “hostfile”. Your inventory can specify information like IP address for each managed node. An inventory can also organize managed nodes, creating and nesting groups for easier scaling. To learn more about inventory, see the Working with Inventory section.
Managed nodes
The network devices, servers, or both that you manage with Ansible. Managed nodes are also sometimes called “hosts”. Ansible is not installed on managed nodes.

5.3. Installing RHEL System Roles in your system

To use RHEL System Roles, install the required packages.

Prerequisites

Procedure

  1. Install the rhel-system-roles package on the system that you want to use as a control node:

    # yum install rhel-system-roles

    If you do not have a Red Hat Ansible Engine Subscription, you can use a limited supported version of Red Hat Ansible Engine provided with your Red Hat Enterprise Linux subscription. In this case, follow these steps:

    1. Enable the RHEL Ansible Engine repository:

      # subscription-manager refresh
      
      # subscription-manager repos --enable ansible-2-for-rhel-8-x86_64-rpms
    2. Install Ansible Engine:

      # yum install ansible

As a result, you are able to create an Ansible playbook.

Additional resources

5.4. Applying a role

The following procedure describes how to apply a particular role.

Prerequisites

  • Ensure that the rhel-system-roles package is installed on the system that you want to use as a control node:

    # yum install rhel-system-roles
  • You need the ansible package to run playbooks that use RHEL System Roles. Ensure that the Ansible Engine repository is enabled, and the ansible package is installed on the system that you want to use as a control node.

    • If you do not have a Red Hat Ansible Engine Subscription, you can use a limited supported version of Red Hat Ansible Engine provided with your Red Hat Enterprise Linux subscription. In this case, follow these steps:

      1. Enable the RHEL Ansible Engine repository:

        # subscription-manager refresh
        # subscription-manager repos --enable ansible-2-for-rhel-8-x86_64-rpms
      2. Install Ansible Engine:

        # yum install ansible
    • If you have a Red Hat Ansible Engine Subscription, follow the procedure described in How do I Download and Install Red Hat Ansible Engine?.
  • Ensure that you are able to create an Ansible inventory.

    Inventories represent the hosts, host groups, and some of the configuration parameters used by the Ansible playbooks.

    Playbooks are typically human-readable, and are defined in ini, yaml, json, and other file formats.

  • Ensure that you are able to create an Ansible playbook.

    Playbooks represent Ansible’s configuration, deployment, and orchestration language. By using playbooks, you can declare and manage configurations of remote machines, deploy multiple remote machines or orchestrate steps of any manual ordered process.

    A playbook is a list of one or more plays. Every play can include Ansible variables, tasks, or roles.

    Playbooks are human-readable, and are defined in the yaml format.

Procedure

  1. Create the required Ansible inventory containing the hosts and groups that you want to manage. Here is an example using a file called inventory.ini of a group of hosts called webservers:

    [webservers]
    host1
    host2
    host3
  2. Create an Ansible playbook including the required role. The following example shows how to use roles through the roles: option for a playbook:

    The following example shows how to use roles through the roles: option for a given play:

    ---
    - hosts: webservers
      roles:
         - rhel-system-roles.network
         - rhel-system-roles.timesync
    Note

    Every role includes a README file, which documents how to use the role and supported parameter values. You can also find an example playbook for a particular role under the documentation directory of the role. Such documentation directory is provided by default with the rhel-system-roles package, and can be found in the following location:

    /usr/share/doc/rhel-system-roles/SUBSYSTEM/

    Replace SUBSYSTEM with the name of the required role, such as selinux, kdump, network, timesync, or storage.

  3. To execute the playbook on specific hosts, you must perform one of the following:

    • Edit the playbook to use hosts: host1[,host2,…​], or hosts: all, and execute the command:

      # ansible-playbook name.of.the.playbook
    • Edit the inventory to ensure that the hosts you want to use are defined in a group, and execute the command:

      # ansible-playbook -i name.of.the.inventory name.of.the.playbook
    • Specify all hosts when executing the ansible-playbook command:

      # ansible-playbook -i host1,host2,... name.of.the.playbook
      Important

      Be aware that the -i flag specifies the inventory of all hosts that are available. If you have multiple targeted hosts, but want to select a host against which you want to run the playbook, you can add a variable in the playbook to be able to select a host. For example:

      Ansible Playbook | example-playbook.yml:
      
      - hosts: "{{ target_host }}"
        roles:
           - rhel-system-roles.network
           - rhel-system-roles.timesync

      Playbook execution command:

      # ansible-playbook -i host1,..hostn -e target_host=host5 example-playbook.yml

5.5. Introduction to the metrics System Role

RHEL System Roles is a collection of Ansible roles and modules that provide a consistent configuration interface to remotely manage multiple RHEL systems. The metrics System Role configures performance analysis services for the local system and, optionally, includes a list of remote systems to be monitored by the local system. The metrics System Role enables you to use pcp to monitor your systems performance without having to configure pcp separately, as the set-up and deployment of pcp is handled by the playbook.

Table 5.1. Metrics system role variables

Role variableDescriptionExample usage

metrics_monitored_hosts

List of remote hosts to be analyzed by the target host. These hosts will have metrics recorded on the target host, so ensure enough disk space exists below /var/log for each host.

metrics_monitored_hosts: ["webserver.example.com", "database.example.com"]

metrics_retention_days

Configures the number of days for performance data retention before deletion.

metrics_retention_days: 14

metrics_graph_service

A boolean flag that enables the host to be set up with services for performance data visualization via pcp and grafana. Set to false by default.

metrics_graph_service: false

metrics_query_service

A boolean flag that enables the host to be set up with time series query services for querying recorded pcp metrics via redis. Set to false by default.

metrics_query_service: false

metrics_provider

Specifies which metrics collector to use to provide metrics. Currently, pcp is the only supported metrics provider.

metrics_provider: "pcp"

Note

For details about the parameters used in metrics_connections and additional information about the metrics System Role, see the /usr/share/ansible/roles/rhel-system-roles.metrics/README.md file.

5.6. Using the metrics System Role to monitor your local system with visualization

This procedure describes how to use the metrics RHEL System Role to monitor your local system while simultaneously provisioning data visualization via grafana.

Prerequisites

  • You have Red Hat Ansible Engine installed on the machine you want to monitor.
  • You have the rhel-system-roles package installed on the machine you want to monitor.

Procedure

  1. Configure localhost in the the /etc/ansible/hosts Ansible inventory by adding the following content to the inventory:

    localhost ansible_connection=local
  2. Create an Ansible playbook with the following content:

    ---
    - hosts: localhost
      vars:
        metrics_graph_service: yes
      roles:
        - rhel-system-roles.metrics
  3. Run the Ansible playbook:

    # ansible-playbook name_of_your_playbook.yml
    Note

    Since the metrics_graph_service boolean is set to value="yes", grafana is automatically installed and provisioned with pcp added as a data source.

  4. To view visualization of the metrics being collected on your machine, access the grafana web interface as described in Accessing the Grafana web UI.

5.7. Using the metrics System Role to setup a fleet of individual systems to monitor themselves

This procedure describes how to use the metrics System Role to set up a fleet of machines to monitor themselves.

Prerequisites

  • You have Red Hat Ansible Engine installed on the machine you want to use to run the playbook.
  • You have the rhel-system-roles package installed on the machine you want to use to run the playbook.

Procedure

  1. Add the name or IP of the machines you wish to monitor via the playbook to the /etc/ansible/hosts Ansible inventory file under an identifying group name enclosed in brackets:

    [remotes]
    webserver.example.com
    database.example.com
  2. Create an Ansible playbook with the following content:

    ---
    - hosts: remotes
      vars:
        metrics_retention_days: 0
      roles:
        - rhel-system-roles.metrics
  3. Run the Ansible playbook:

    # ansible-playbook name_of_your_playbook.yml

5.8. Using the metrics System Role to monitor a fleet of machines centrally via your local machine

This procedure describes how to use the metrics System Role to set up your local machine to centrally monitor a fleet of machines while also provisioning visualization of the data via grafana and querying of the data via redis.

Prerequisites

  • You have Red Hat Ansible Engine installed on the machine you want to use to run the playbook.
  • You have the rhel-system-roles package installed on the machine you want to use to run the playbook.

Procedure

  1. Create an Ansible playbook with the following content:

    ---
    - hosts: localhost
      vars:
        metrics_graph_service: yes
        metrics_query_service: yes
        metrics_retention_days: 10
        metrics_monitored_hosts: ["database.example.com", "webserver.example.com"]
      roles:
        - rhel-system-roles.metrics
  2. Run the Ansible playbook:

    # ansible-playbook name_of_your_playbook.yml
    Note

    Since the metrics_graph_service and metrics_query_service booleans are set to value="yes", grafana is automatically installed and provisioned with pcp added as a data source with the pcp data recording indexed into redis, allowing the pcp querying language to be used for complex querying of the data.

  3. To view graphical representation of the metrics being collected centrally by your machine and to query the data, access the grafana web interface as described in Accessing the Grafana web UI.

5.9. Setting up authentication while monitoring a system using the metrics System Role

PCP supports the scram-sha-256 authentication mechanism through the Simple Authentication Security Layer (SASL) framework. The metrics RHEL System Role automates the steps to setup authentication using the scram-sha-256 authentication mechanism. This procedure describes how to setup authentication using the metrics RHEL System Role.

Prerequisites

  • You have Red Hat Ansible Engine installed on the machine you want to use to run the playbook.
  • You have the rhel-system-roles package installed on the machine you want to use to run the playbook.

Procedure

  1. Include the following variables in the Ansible playbook you want to setup authentication for:

    ---
      vars:
        metrics_username: your_username
        metrics_password: your_password
  2. Run the Ansible playbook:

    # ansible-playbook name_of_your_playbook.yml

Verification steps

  • Verify the sasl configuration:

    # pminfo -f -h "pcp://127.0.0.1?username=your_username" disk.dev.read
    Password:
    disk.dev.read
    inst [0 or "sda"] value 19540

5.10. Using the metrics System Role to configure and enable metrics collection for SQL Server

This procedure describes how to use the metrics RHEL System Role to automate the configuration and enabling of metrics collection for Microsoft SQL Server via pcp on your local system.

Prerequisites

  • You have Red Hat Ansible Engine installed on the machine you want to monitor.
  • You have the rhel-system-roles package installed on the machine you want to monitor.
  • You have installed Microsoft SQL Server for Red Hat Enterprise Linux and established a 'trusted' connection to an SQL server.
  • You have installed the Microsoft ODBC driver for SQL Server for Red Hat Enterprise Linux.

Procedure

  1. Configure localhost in the the /etc/ansible/hosts Ansible inventory by adding the following content to the inventory:

    localhost ansible_connection=local
  2. Create an Ansible playbook that contains the following content:

    ---
    - hosts: localhost
      roles:
        - role: rhel-system-roles.metrics
          vars:
            metrics_from_sql: yes
  3. Run the Ansible playbook:

    # ansible-playbook name_of_your_playbook.yml

Verification steps

  • Use the pcp command to verify that SQL Server PMDA agent (mssql) is loaded and running:

    # pcp
    platform: Linux rhel82-2.local 4.18.0-167.el8.x86_64 #1 SMP Sun Dec 15 01:24:23 UTC 2019 x86_64
     hardware: 2 cpus, 1 disk, 1 node, 2770MB RAM
     timezone: PDT+7
     services: pmcd pmproxy
         pmcd: Version 5.0.2-1, 12 agents, 4 clients
         pmda: root pmcd proc pmproxy xfs linux nfsclient mmv kvm mssql
               jbd2 dm
     pmlogger: primary logger: /var/log/pcp/pmlogger/rhel82-2.local/20200326.16.31
         pmie: primary engine: /var/log/pcp/pmie/rhel82-2.local/pmie.log


[1] This documentation is installed automatically with the rhel-system-roles package.