Menu Close
Settings Close

Language and Page Formatting Options

Red Hat Training

A Red Hat training course is available for RHEL 8

Chapter 5. Monitoring performance using RHEL System Roles

As a system administrator, you can use the Metrics RHEL System Role with any Ansible Automation Platform control node to monitor the performance of a system.

5.1. Preparing a control node and managed nodes to use RHEL System Roles

Before you can use individual RHEL System Roles to manage services and settings, prepare the involved hosts.

5.1.1. Introduction to RHEL System Roles

RHEL System Roles is a collection of Ansible roles and modules. RHEL System Roles provide a configuration interface to remotely manage multiple RHEL systems. The interface enables managing system configurations across multiple versions of RHEL, as well as adopting new major releases.

On Red Hat Enterprise Linux 8, the interface currently consists of the following roles:

  • Certificate Issuance and Renewal (certificate)
  • Cockpit (cockpit)
  • Firewalld (firewall)
  • HA Cluster (ha_cluster)
  • Kernel Dumps (kdump)
  • Kernel Settings (kernel_settings)
  • Logging (logging)
  • Metrics (PCP) (metrics)
  • Microsoft SQL Server (microsoft.sql.server)
  • Networking (network)
  • Network Bound Disk Encryption client and Network Bound Disk Encryption server (nbde_client and nbde_server)
  • Postfix (postfix)
  • SELinux (selinux)
  • SSH client (ssh)
  • SSH server (sshd)
  • Storage (storage)
  • Terminal Session Recording (tlog)
  • Time Synchronization (timesync)
  • VPN (vpn)

All these roles are provided by the rhel-system-roles package available in the AppStream repository.

Additional resources

5.1.2. RHEL System Roles terminology

You can find the following terms across this documentation:

Ansible playbook
Playbooks are Ansible’s configuration, deployment, and orchestration language. They can describe a policy you want your remote systems to enforce, or a set of steps in a general IT process.
Control node
Any machine with Ansible installed. You can run commands and playbooks, invoking /usr/bin/ansible or /usr/bin/ansible-playbook, from any control node. You can use any computer that has Python installed on it as a control node - laptops, shared desktops, and servers can all run Ansible. However, you cannot use a Windows machine as a control node. You can have multiple control nodes.
Inventory
A list of managed nodes. An inventory file is also sometimes called a “hostfile”. Your inventory can specify information like IP address for each managed node. An inventory can also organize managed nodes, creating and nesting groups for easier scaling. To learn more about inventory, see the Working with Inventory section.
Managed nodes
The network devices, servers, or both that you manage with Ansible. Managed nodes are also sometimes called “hosts”. Ansible is not installed on managed nodes.

5.1.3. Preparing a control node

RHEL includes Ansible Core in the AppStream repository with a limited scope of support. If you require additional support for Ansible, contact Red Hat to learn more about the Ansible Automation Platform subscription.

Important

On RHEL 8.6 and later, use Ansible Core if you do not have a Red Hat Ansible Automation Platform subscription. Do not install Ansible Engine from the ansible-2-for-rhel-8-x86_64-rpms repository. Packages in this repository might not be compatible with Ansible automation content in RHEL 8.6 and later. Additionally, Red Hat does not provide security updates and bug fixes for Ansible Engine for the same amount of time as for RHEL. For further details, see Using Ansible in RHEL 8.6 and later.

Prerequisites

  • You installed RHEL 8.6 or later.
  • You registered the system to the Customer Portal.
  • You attached a Red Hat Enterprise Linux Server subscription to the system.
  • If available in your Customer Portal account, you attached an Ansible Automation Platform subscription to the system.

Procedure

  1. Install the rhel-system-roles package:

    [root@control-node]# yum install rhel-system-roles

    This command installs Ansible Core as a dependency.

  2. Create a user that you later use to manage and execute playbooks:

    [root@control-node]# useradd ansible
  3. Switch to the newly created ansible user:

    [root@control-node]# su - ansible

    Perform the rest of the procedure as this user.

  4. Create an SSH public and private key

    [ansible@control-node]$ ssh-keygen
    Generating public/private rsa key pair.
    Enter file in which to save the key (/home/ansible/.ssh/id_rsa): password
    ...

    Use the suggested default location for the key file.

  5. Optional: Configure an SSH agent to prevent Ansible from prompting you for the SSH key password each time you establish a connection.
  6. Create the ~/.ansible.cfg file with the following content:

    [defaults]
    inventory = /home/ansible/inventory
    remote_user = ansible
    
    [privilege_escalation]
    become = True
    become_method = sudo
    become_user = root
    become_ask_pass = True

    With these settings:

    • Ansible manages hosts in the specified inventory file.
    • Ansible uses the account set in the remote_user parameter when it establishes SSH connections to managed nodes.
    • Ansible uses the sudo utility to execute tasks on managed nodes as the root user.

      For security reasons, configure sudo on managed nodes to require entering the password of the remote user to become root. By specifying the become_ask_pass=True setting in ~/.ansible.cfg, Ansible prompts for this password when you execute a playbook.

    Settings in the ~/.ansible.cfg file have a higher priority and override settings from the global /etc/ansible/ansible.cfg file.

  7. Create the ~/inventory file. For example, the following is an inventory file in the INI format with three hosts and one host group named US:

    managed-node-01.example.com
    
    [US]
    managed-node-02.example.com ansible_host=192.0.2.100
    managed-node-03.example.com

    Note that the control node must be able to resolve the hostnames. If the DNS server cannot resolve certain hostnames, add the ansible_host parameter next to the host entry to specify its IP address.

5.1.4. Preparing a managed node

Ansible does not use an agent on managed hosts. The only requirements are Python, which is installed by default on RHEL, and SSH access to the managed host.

However, direct SSH access as the root user can be a security risk. Therefore, when you prepare a managed node, you create a local user on this node and configure a sudo policy. Ansible on the control node can then use this account to log in to the managed node and execute playbooks as different users, such as root.

Prerequisites

  • You prepared the control node.

Procedure

  1. Create a user:

    [root@managed-node-01]# useradd ansible

    The control node later uses this user to establish an SSH connection to this host.

  2. Set a password to the ansible user:

    [root@managed-node-01]# passwd ansible
    Changing password for user ansible.
    New password: password
    Retype new password: password
    passwd: all authentication tokens updated successfully.

    You must enter this password when Ansible uses sudo to perform tasks as the root user.

  3. Install the ansible user’s SSH public key on the managed node:

    1. Log into the control node as the ansible user, and copy the SSH public key to the managed node:

      [ansible@control-node]$ ssh-copy-id managed-node-01.example.com
      /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/ansible/.ssh/id_rsa.pub"
      The authenticity of host 'managed-node-01.example.com (192.0.2.100)' can't be established.
      ECDSA key fingerprint is SHA256:9bZ33GJNODK3zbNhybokN/6Mq7hu3vpBXDrCxe7NAvo.
      Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
      /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
      /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
      ansible@managed-node-01.example.com's password: password
      
      Number of key(s) added: 1
      
      Now try logging into the machine, with:   "ssh 'managed-node-01.example.com'"
      and check to make sure that only the key(s) you wanted were added.
    2. Remotely execute a command on the control node to verify the SSH connection:

      [ansible@control-node]$ ssh managed-node-01.example.com whoami
      ansible
  4. Create a sudo configuration for the ansible user:

    1. Use the visudo command to create and edit the /etc/sudoers.d/ansible file:

      [root@managed-node-01]# visudo /etc/sudoers.d/ansible

      The benefit of using visudo over a normal editor is that this utility provides basic sanity checks and checks for parse errors before installing the file.

    2. Enter the following content to the /etc/sudoers.d/ansible file:

      ansible ALL=(ALL) NOPASSWD: ALL

      These settings grant permissions to the ansible user to run all commands as any user and group on this host without entering the password of the ansible user.

Additional resources

5.1.5. Verifying access from the control node to managed nodes

After you configured the control node and prepared managed nodes, test that Ansible can connect to the managed nodes.

Perform this procedure as the ansible user on the control node.

Prerequisites

  • You prepared the control node as described in Preparing a control node.
  • You prepared at least one managed node as described in Preparing a managed node.
  • If you want to run playbooks on host groups, the managed node is listed in the inventory file on the control node.

Procedure

  1. Use the Ansible ping module to verify that you can execute commands on an all managed hosts:

    [ansible@control-node]$ ansible all -m ping
    BECOME password: password
    managed-node-01.example.com | SUCCESS => {
        "ansible_facts": {
            "discovered_interpreter_python": "/usr/bin/python3"
        },
        "changed": false,
        "ping": "pong"
    }
    ...

    The hard-coded all host group dynamically contains all hosts listed in the inventory file.

  2. Use the Ansible command module to run the whoami utility on a managed host:

    [ansible@control-node]$ ansible managed-node-01.example.com -m command -a whoami
    BECOME password: password
    managed-node-01.example.com | CHANGED | rc=0 >>
    root

    If the command returns root, you configured sudo on the managed nodes correctly, and privilege escalation works.

5.2. Introduction to the Metrics System Role

RHEL System Roles is a collection of Ansible roles and modules that provide a consistent configuration interface to remotely manage multiple RHEL systems. The Metrics System Role configures performance analysis services for the local system and, optionally, includes a list of remote systems to be monitored by the local system. The Metrics System Role enables you to use pcp to monitor your systems performance without having to configure pcp separately, as the set-up and deployment of pcp is handled by the playbook.

Table 5.1. Metrics system role variables

Role variableDescriptionExample usage

metrics_monitored_hosts

List of remote hosts to be analyzed by the target host. These hosts will have metrics recorded on the target host, so ensure enough disk space exists below /var/log for each host.

metrics_monitored_hosts: ["webserver.example.com", "database.example.com"]

metrics_retention_days

Configures the number of days for performance data retention before deletion.

metrics_retention_days: 14

metrics_graph_service

A boolean flag that enables the host to be set up with services for performance data visualization via pcp and grafana. Set to false by default.

metrics_graph_service: no

metrics_query_service

A boolean flag that enables the host to be set up with time series query services for querying recorded pcp metrics via redis. Set to false by default.

metrics_query_service: no

metrics_provider

Specifies which metrics collector to use to provide metrics. Currently, pcp is the only supported metrics provider.

metrics_provider: "pcp"

Note

For details about the parameters used in metrics_connections and additional information about the Metrics System Role, see the /usr/share/ansible/roles/rhel-system-roles.metrics/README.md file.

5.3. Using the Metrics System Role to monitor your local system with visualization

This procedure describes how to use the Metrics RHEL System Role to monitor your local system while simultaneously provisioning data visualization via Grafana.

Prerequisites

  • The Ansible Core package is installed on the control machine.
  • You have the rhel-system-roles package installed on the machine you want to monitor.

Procedure

  1. Configure localhost in the /etc/ansible/hosts Ansible inventory by adding the following content to the inventory:

    localhost ansible_connection=local
  2. Create an Ansible playbook with the following content:

    ---
    - hosts: localhost
      vars:
        metrics_graph_service: yes
      roles:
        - rhel-system-roles.metrics
  3. Run the Ansible playbook:

    # ansible-playbook name_of_your_playbook.yml
    Note

    Since the metrics_graph_service boolean is set to value="yes", Grafana is automatically installed and provisioned with pcp added as a data source.

  4. To view visualization of the metrics being collected on your machine, access the grafana web interface as described in Accessing the Grafana web UI.

5.4. Using the Metrics System Role to setup a fleet of individual systems to monitor themselves

This procedure describes how to use the Metrics System Role to set up a fleet of machines to monitor themselves.

Prerequisites

  • The Ansible Core package is installed on the control machine.
  • You have the rhel-system-roles package installed on the machine you want to use to run the playbook.
  • You have the SSH connection established.

Procedure

  1. Add the name or IP of the machines you wish to monitor via the playbook to the /etc/ansible/hosts Ansible inventory file under an identifying group name enclosed in brackets:

    [remotes]
    webserver.example.com
    database.example.com
  2. Create an Ansible playbook with the following content:

    ---
    - hosts: remotes
      vars:
        metrics_retention_days: 0
      roles:
        - rhel-system-roles.metrics
  3. Run the Ansible playbook:

    # ansible-playbook name_of_your_playbook.yml -k

Where the -k prompt for password to connect to remote system.

5.5. Using the Metrics System Role to monitor a fleet of machines centrally via your local machine

This procedure describes how to use the Metrics System Role to set up your local machine to centrally monitor a fleet of machines while also provisioning visualization of the data via grafana and querying of the data via redis.

Prerequisites

  • The Ansible Core package is installed on the control machine.
  • You have the rhel-system-roles package installed on the machine you want to use to run the playbook.

Procedure

  1. Create an Ansible playbook with the following content:

    ---
    - hosts: localhost
      vars:
        metrics_graph_service: yes
        metrics_query_service: yes
        metrics_retention_days: 10
        metrics_monitored_hosts: ["database.example.com", "webserver.example.com"]
      roles:
        - rhel-system-roles.metrics
  2. Run the Ansible playbook:

    # ansible-playbook name_of_your_playbook.yml
    Note

    Since the metrics_graph_service and metrics_query_service booleans are set to value="yes", grafana is automatically installed and provisioned with pcp added as a data source with the pcp data recording indexed into redis, allowing the pcp querying language to be used for complex querying of the data.

  3. To view graphical representation of the metrics being collected centrally by your machine and to query the data, access the grafana web interface as described in Accessing the Grafana web UI.

5.6. Setting up authentication while monitoring a system using the Metrics System Role

PCP supports the scram-sha-256 authentication mechanism through the Simple Authentication Security Layer (SASL) framework. The Metrics RHEL System Role automates the steps to setup authentication using the scram-sha-256 authentication mechanism. This procedure describes how to setup authentication using the Metrics RHEL System Role.

Prerequisites

  • The Ansible Core package is installed on the control machine.
  • You have the rhel-system-roles package installed on the machine you want to use to run the playbook.

Procedure

  1. Include the following variables in the Ansible playbook you want to setup authentication for:

    ---
      vars:
        metrics_username: your_username
        metrics_password: your_password
  2. Run the Ansible playbook:

    # ansible-playbook name_of_your_playbook.yml

Verification steps

  • Verify the sasl configuration:

    # pminfo -f -h "pcp://ip_adress?username=your_username" disk.dev.read
    Password:
    disk.dev.read
    inst [0 or "sda"] value 19540

    ip_adress should be replaced by the IP address of the host.

5.7. Using the Metrics System Role to configure and enable metrics collection for SQL Server

This procedure describes how to use the Metrics RHEL System Role to automate the configuration and enabling of metrics collection for Microsoft SQL Server via pcp on your local system.

Prerequisites

  • The Ansible Core package is installed on the control machine.
  • You have the rhel-system-roles package installed on the machine you want to monitor.
  • You have installed Microsoft SQL Server for Red Hat Enterprise Linux and established a 'trusted' connection to an SQL server. See Install SQL Server and create a database on Red Hat.
  • You have installed the Microsoft ODBC driver for SQL Server for Red Hat Enterprise Linux. See Red Hat Enterprise Server and Oracle Linux.

Procedure

  1. Configure localhost in the /etc/ansible/hosts Ansible inventory by adding the following content to the inventory:

    localhost ansible_connection=local
  2. Create an Ansible playbook that contains the following content:

    ---
    - hosts: localhost
      roles:
        - role: rhel-system-roles.metrics
          vars:
            metrics_from_mssql: yes
  3. Run the Ansible playbook:

    # ansible-playbook name_of_your_playbook.yml

Verification steps

  • Use the pcp command to verify that SQL Server PMDA agent (mssql) is loaded and running:

    # pcp
    platform: Linux rhel82-2.local 4.18.0-167.el8.x86_64 #1 SMP Sun Dec 15 01:24:23 UTC 2019 x86_64
     hardware: 2 cpus, 1 disk, 1 node, 2770MB RAM
     timezone: PDT+7
     services: pmcd pmproxy
         pmcd: Version 5.0.2-1, 12 agents, 4 clients
         pmda: root pmcd proc pmproxy xfs linux nfsclient mmv kvm mssql
               jbd2 dm
     pmlogger: primary logger: /var/log/pcp/pmlogger/rhel82-2.local/20200326.16.31
         pmie: primary engine: /var/log/pcp/pmie/rhel82-2.local/pmie.log