Red Hat Training

A Red Hat training course is available for RHEL 8

Chapter 5. Monitoring performance using RHEL System Roles

As a system administrator, you can use the metrics RHEL System Role with any Ansible Automation Platform control node to monitor the performance of a system.

5.1. Preparing a control node and managed nodes to use RHEL System Roles

Before you can use individual RHEL System Roles to manage services and settings, you must prepare the control node and managed nodes.

5.1.1. Preparing a control node on RHEL 8

Before using RHEL System Roles, you must configure a control node. This system then configures the managed hosts from the inventory according to the playbooks.

Prerequisites

Procedure

  1. Install the rhel-system-roles package:

    [root@control-node]# yum install rhel-system-roles

    This command installs the ansible-core package as a dependency.

  2. Create a user named ansible to manage and run playbooks:

    [root@control-node]# useradd ansible
  3. Switch to the newly created ansible user:

    [root@control-node]# su - ansible

    Perform the rest of the procedure as this user.

  4. Create an SSH public and private key:

    [ansible@control-node]$ ssh-keygen
    Generating public/private rsa key pair.
    Enter file in which to save the key (/home/ansible/.ssh/id_rsa):
    Enter passphrase (empty for no passphrase): <password>
    Enter same passphrase again: <password>
    ...

    Use the suggested default location for the key file.

  5. Optional: To prevent Ansible from prompting you for the SSH key password each time you establish a connection, configure an SSH agent.
  6. Create the ~/.ansible.cfg file with the following content:

    [defaults]
    inventory = /home/ansible/inventory
    remote_user = ansible
    
    [privilege_escalation]
    become = True
    become_method = sudo
    become_user = root
    become_ask_pass = True
    Note

    Settings in the ~/.ansible.cfg file have a higher priority and override settings from the global /etc/ansible/ansible.cfg file.

    With these settings, Ansible performs the following actions:

    • Manages hosts in the specified inventory file.
    • Uses the account set in the remote_user parameter when it establishes SSH connections to managed nodes.
    • Uses the sudo utility to execute tasks on managed nodes as the root user.
    • Prompts for the root password of the remote user every time you apply a playbook. This is recommended for security reasons.
  7. Create an ~/inventory file in INI or YAML format that lists the hostnames of managed hosts. You can also define groups of hosts in the inventory file. For example, the following is an inventory file in the INI format with three hosts and one host group named US:

    managed-node-01.example.com
    
    [US]
    managed-node-02.example.com ansible_host=192.0.2.100
    managed-node-03.example.com

    Note that the control node must be able to resolve the hostnames. If the DNS server cannot resolve certain hostnames, add the ansible_host parameter next to the host entry to specify its IP address.

Next steps

5.1.2. Preparing a managed node

Managed nodes are the systems listed in the inventory and which will be configured by the control node according to the playbook. You do not have to install Ansible on managed hosts.

Prerequisites

  • You prepared the control node. For more information, see Preparing a control node on RHEL 8.
  • You have SSH access from the control node.

    Important

    Direct SSH access as the root user is a security risk. To reduce this risk, you will create a local user on this node and configure a sudo policy when preparing a managed node. Ansible on the control node can then use the local user account to log in to the managed node and run playbooks as different users, such as root.

Procedure

  1. Create a user named ansible:

    [root@managed-node-01]# useradd ansible

    The control node later uses this user to establish an SSH connection to this host.

  2. Set a password for the ansible user:

    [root@managed-node-01]# passwd ansible
    Changing password for user ansible.
    New password: <password>
    Retype new password: <password>
    passwd: all authentication tokens updated successfully.

    You must enter this password when Ansible uses sudo to perform tasks as the root user.

  3. Install the ansible user’s SSH public key on the managed node:

    1. Log in to the control node as the ansible user, and copy the SSH public key to the managed node:

      [ansible@control-node]$ ssh-copy-id managed-node-01.example.com
      /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/ansible/.ssh/id_rsa.pub"
      The authenticity of host 'managed-node-01.example.com (192.0.2.100)' can't be established.
      ECDSA key fingerprint is SHA256:9bZ33GJNODK3zbNhybokN/6Mq7hu3vpBXDrCxe7NAvo.
    2. When prompted, connect by entering yes:

      Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
      /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
      /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
    3. When prompted, enter the password:

      ansible@managed-node-01.example.com's password: <password>
      
      Number of key(s) added: 1
      
      Now try logging into the machine, with:   "ssh '<managed-node-01.example.com>'"
      and check to make sure that only the key(s) you wanted were added.
    4. Verify the SSH connection by remotely executing a command on the control node:

      [ansible@control-node]$ ssh <managed-node-01.example.com> whoami
      ansible
  4. Create a sudo configuration for the ansible user:

    1. Create and edit the /etc/sudoers.d/ansible file by using the visudo command:

      [root@managed-node-01]# visudo /etc/sudoers.d/ansible

      The benefit of using visudo over a normal editor is that this utility provides basic sanity checks and checks for parse errors before installing the file.

    2. Configure a sudoers policy in the /etc/sudoers.d/ansible file that meets your requirements, for example:

      • To grant permissions to the ansible user to run all commands as any user and group on this host after entering the ansible user’s password, use:

        ansible ALL=(ALL) ALL
      • To grant permissions to the ansible user to run all commands as any user and group on this host without entering the ansible user’s password, use:

        ansible ALL=(ALL) NOPASSWD: ALL

    Alternatively, configure a more fine-granular policy that matches your security requirements. For further details on sudoers policies, see the sudoers(5) man page.

Verification

  1. Verify that you can execute commands from the control node on an all managed nodes:

    [ansible@control-node]$ ansible all -m ping
    BECOME password: <password>
    managed-node-01.example.com | SUCCESS => {
        	"ansible_facts": {
        	    "discovered_interpreter_python": "/usr/bin/python3"
        	},
        	"changed": false,
        	"ping": "pong"
    }
    ...

    The hard-coded all group dynamically contains all hosts listed in the inventory file.

  2. Verify that privilege escalation works correctly by running the whoami utility on a managed host by using the Ansible command module:

    [ansible@control-node]$ ansible managed-node-01.example.com -m command -a whoami
    BECOME password: <password>
    managed-node-01.example.com | CHANGED | rc=0 >>
    root

    If the command returns root, you configured sudo on the managed nodes correctly.

Additional resources

5.2. Introduction to the metrics System Role

RHEL System Roles is a collection of Ansible roles and modules that provide a consistent configuration interface to remotely manage multiple RHEL systems. The metrics System Role configures performance analysis services for the local system and, optionally, includes a list of remote systems to be monitored by the local system. The metrics System Role enables you to use pcp to monitor your systems performance without having to configure pcp separately, as the set-up and deployment of pcp is handled by the playbook.

Table 5.1. metrics system role variables

Role variableDescriptionExample usage

metrics_monitored_hosts

List of remote hosts to be analyzed by the target host. These hosts will have metrics recorded on the target host, so ensure enough disk space exists below /var/log for each host.

metrics_monitored_hosts: ["webserver.example.com", "database.example.com"]

metrics_retention_days

Configures the number of days for performance data retention before deletion.

metrics_retention_days: 14

metrics_graph_service

A boolean flag that enables the host to be set up with services for performance data visualization via pcp and grafana. Set to false by default.

metrics_graph_service: no

metrics_query_service

A boolean flag that enables the host to be set up with time series query services for querying recorded pcp metrics via redis. Set to false by default.

metrics_query_service: no

metrics_provider

Specifies which metrics collector to use to provide metrics. Currently, pcp is the only supported metrics provider.

metrics_provider: "pcp"

metrics_manage_firewall

Uses the firewall role to manage port access directly from the metrics role. Set to false by default.

metrics_manage_firewall: true

metrics_manage_selinux

Uses the selinux role to manage port access directly from the metrics role. Set to false by default.

metrics_manage_selinux: true

Note

For details about the parameters used in metrics_connections and additional information about the metrics System Role, see the /usr/share/ansible/roles/rhel-system-roles.metrics/README.md file.

5.3. Using the metrics System Role to monitor your local system with visualization

This procedure describes how to use the metrics RHEL System Role to monitor your local system while simultaneously provisioning data visualization via Grafana.

Prerequisites

  • The Ansible Core package is installed on the control machine.
  • You have the rhel-system-roles package installed on the machine you want to monitor.

Procedure

  1. Configure localhost in the /etc/ansible/hosts Ansible inventory by adding the following content to the inventory:

    localhost ansible_connection=local
  2. Create an Ansible playbook with the following content:

    ---
    - name: Manage metrics
      hosts: localhost
      vars:
        metrics_graph_service: yes
        metrics_manage_firewall: true
        metrics_manage_selinux: true
      roles:
        - rhel-system-roles.metrics
  3. Run the Ansible playbook:

    # ansible-playbook name_of_your_playbook.yml
    Note

    Because the metrics_graph_service boolean is set to value="yes", Grafana is automatically installed and provisioned with pcp added as a data source. Because metrics_manage_firewall and metrics_manage_selinux are both set to true, the metrics role uses the firewall and selinux system roles to manage the ports used by the metrics role.

  4. To view visualization of the metrics being collected on your machine, access the grafana web interface as described in Accessing the Grafana web UI.

5.4. Using the metrics System Role to set up a fleet of individual systems to monitor themselves

This procedure describes how to use the metrics System Role to set up a fleet of machines to monitor themselves.

Prerequisites

  • The Ansible Core package is installed on the control machine.
  • You have the rhel-system-roles package installed on the machine you want to use to run the playbook.
  • You have the SSH connection established.

Procedure

  1. Add the name or IP address of the machines you want to monitor via the playbook to the /etc/ansible/hosts Ansible inventory file under an identifying group name enclosed in brackets:

    [remotes]
    webserver.example.com
    database.example.com
  2. Create an Ansible playbook with the following content:

    ---
    - hosts: remotes
      vars:
        metrics_retention_days: 0
        metrics_manage_firewall: true
        metrics_manage_selinux: true
      roles:
        - rhel-system-roles.metrics
    Note

    Because metrics_manage_firewall and metrics_manage_selinux are both set to true, the metrics role uses the firewall and selinux roles to manage the ports used by the metrics role.

  3. Run the Ansible playbook:

    # ansible-playbook name_of_your_playbook.yml -k

    Where the -k prompt for password to connect to remote system.

5.5. Using the metrics System Role to monitor a fleet of machines centrally via your local machine

This procedure describes how to use the metrics System Role to set up your local machine to centrally monitor a fleet of machines while also provisioning visualization of the data via grafana and querying of the data via redis.

Prerequisites

  • The Ansible Core package is installed on the control machine.
  • You have the rhel-system-roles package installed on the machine you want to use to run the playbook.

Procedure

  1. Create an Ansible playbook with the following content:

    ---
    - hosts: localhost
      vars:
        metrics_graph_service: yes
        metrics_query_service: yes
        metrics_retention_days: 10
        metrics_monitored_hosts: ["database.example.com", "webserver.example.com"]
        metrics_manage_firewall: yes
        metrics_manage_selinux: yes
      roles:
        - rhel-system-roles.metrics
  2. Run the Ansible playbook:

    # ansible-playbook name_of_your_playbook.yml
    Note

    Because the metrics_graph_service and metrics_query_service booleans are set to value="yes", grafana is automatically installed and provisioned with pcp added as a data source with the pcp data recording indexed into redis, allowing the pcp querying language to be used for complex querying of the data. Because metrics_manage_firewall and metrics_manage_selinux are both set to true, the metrics role uses the firewall and selinux roles to manage the ports used by the metrics role.

  3. To view a graphical representation of the metrics being collected centrally by your machine and to query the data, access the grafana web interface as described in Accessing the Grafana web UI.

5.6. Setting up authentication while monitoring a system using the metrics System Role

PCP supports the scram-sha-256 authentication mechanism through the Simple Authentication Security Layer (SASL) framework. The metrics RHEL System Role automates the steps to setup authentication using the scram-sha-256 authentication mechanism. This procedure describes how to setup authentication using the metrics RHEL System Role.

Prerequisites

  • The Ansible Core package is installed on the control machine.
  • You have the rhel-system-roles package installed on the machine you want to use to run the playbook.

Procedure

  1. Include the following variables in the Ansible playbook you want to setup authentication for:

    ---
      vars:
        metrics_username: your_username
        metrics_password: your_password
        metrics_manage_firewall: true
        metrics_manage_selinux: true
    Note

    Because metrics_manage_firewall and metrics_manage_selinux are both set to true, the metrics role uses the firewall and selinux roles to manage the ports used by the metrics role.

  2. Run the Ansible playbook:

    # ansible-playbook name_of_your_playbook.yml

Verification steps

  • Verify the sasl configuration:

    # pminfo -f -h "pcp://ip_adress?username=your_username" disk.dev.read
    Password:
    disk.dev.read
    inst [0 or "sda"] value 19540

    ip_adress should be replaced by the IP address of the host.

5.7. Using the metrics System Role to configure and enable metrics collection for SQL Server

This procedure describes how to use the metrics RHEL System Role to automate the configuration and enabling of metrics collection for Microsoft SQL Server via pcp on your local system.

Prerequisites

  • The Ansible Core package is installed on the control machine.
  • You have the rhel-system-roles package installed on the machine you want to monitor.
  • You have installed Microsoft SQL Server for Red Hat Enterprise Linux and established a 'trusted' connection to an SQL server. See Install SQL Server and create a database on Red Hat.
  • You have installed the Microsoft ODBC driver for SQL Server for Red Hat Enterprise Linux. See Red Hat Enterprise Server and Oracle Linux.

Procedure

  1. Configure localhost in the /etc/ansible/hosts Ansible inventory by adding the following content to the inventory:

    localhost ansible_connection=local
  2. Create an Ansible playbook that contains the following content:

    ---
    - hosts: localhost
      vars:
       metrics_from_mssql: true
       metrics_manage_firewall: true
       metrics_manage_selinux: true
      roles:
       - role: rhel-system-roles.metrics
    Note

    Because metrics_manage_firewall and metrics_manage_selinux are both set to true, the metrics role uses the firewall and selinux roles to manage the ports used by the metrics role.

  3. Run the Ansible playbook:

    # ansible-playbook name_of_your_playbook.yml

Verification steps

  • Use the pcp command to verify that SQL Server PMDA agent (mssql) is loaded and running:

    # pcp
    platform: Linux rhel82-2.local 4.18.0-167.el8.x86_64 #1 SMP Sun Dec 15 01:24:23 UTC 2019 x86_64
     hardware: 2 cpus, 1 disk, 1 node, 2770MB RAM
     timezone: PDT+7
     services: pmcd pmproxy
         pmcd: Version 5.0.2-1, 12 agents, 4 clients
         pmda: root pmcd proc pmproxy xfs linux nfsclient mmv kvm mssql
               jbd2 dm
     pmlogger: primary logger: /var/log/pcp/pmlogger/rhel82-2.local/20200326.16.31
         pmie: primary engine: /var/log/pcp/pmie/rhel82-2.local/pmie.log