Chapter 4. Monitoring performance using RHEL system roles

As a system administrator, you can use the metrics RHEL system role to monitor the performance of a system.

4.1. Preparing a control node and managed nodes to use RHEL system roles

Before you can use individual RHEL system roles to manage services and settings, you must prepare the control node and managed nodes.

4.1.1. Preparing a control node on RHEL 9

Before using RHEL system roles, you must configure a control node. This system then configures the managed hosts from the inventory according to the playbooks.

Prerequisites

  • The system is registered to the Customer Portal.
  • A Red Hat Enterprise Linux Server subscription is attached to the system.
  • Optional: An Ansible Automation Platform subscription is attached to the system.

Procedure

  1. Create a user named ansible to manage and run playbooks:

    [root@control-node]# useradd ansible
  2. Switch to the newly created ansible user:

    [root@control-node]# su - ansible

    Perform the rest of the procedure as this user.

  3. Create an SSH public and private key:

    [ansible@control-node]$ ssh-keygen
    Generating public/private rsa key pair.
    Enter file in which to save the key (/home/ansible/.ssh/id_rsa):
    Enter passphrase (empty for no passphrase): <password>
    Enter same passphrase again: <password>
    ...

    Use the suggested default location for the key file.

  4. Optional: To prevent Ansible from prompting you for the SSH key password each time you establish a connection, configure an SSH agent.
  5. Create the ~/.ansible.cfg file with the following content:

    [defaults]
    inventory = /home/ansible/inventory
    remote_user = ansible
    
    [privilege_escalation]
    become = True
    become_method = sudo
    become_user = root
    become_ask_pass = True
    Note

    Settings in the ~/.ansible.cfg file have a higher priority and override settings from the global /etc/ansible/ansible.cfg file.

    With these settings, Ansible performs the following actions:

    • Manages hosts in the specified inventory file.
    • Uses the account set in the remote_user parameter when it establishes SSH connections to managed nodes.
    • Uses the sudo utility to execute tasks on managed nodes as the root user.
    • Prompts for the root password of the remote user every time you apply a playbook. This is recommended for security reasons.
  6. Create an ~/inventory file in INI or YAML format that lists the hostnames of managed hosts. You can also define groups of hosts in the inventory file. For example, the following is an inventory file in the INI format with three hosts and one host group named US:

    managed-node-01.example.com
    
    [US]
    managed-node-02.example.com ansible_host=192.0.2.100
    managed-node-03.example.com

    Note that the control node must be able to resolve the hostnames. If the DNS server cannot resolve certain hostnames, add the ansible_host parameter next to the host entry to specify its IP address.

  7. Install RHEL system roles:

    • On a RHEL host without Ansible Automation Platform, install the rhel-system-roles package:

      [root@control-node]# dnf install rhel-system-roles

      This command installs the collections in the /usr/share/ansible/collections/ansible_collections/redhat/rhel_system_roles/ directory, and the ansible-core package as a dependency.

    • On Ansible Automation Platform, perform the following steps as the ansible user:

      1. Define Red Hat automation hub as the primary source for content in the ~/.ansible.cfg file.
      2. Install the redhat.rhel_system_roles collection from Red Hat automation hub:

        [ansible@control-node]$ ansible-galaxy collection install redhat.rhel_system_roles

        This command installs the collection in the ~/.ansible/collections/ansible_collections/redhat/rhel_system_roles/ directory.

Next steps

4.1.2. Preparing a managed node

Managed nodes are the systems listed in the inventory and which will be configured by the control node according to the playbook. You do not have to install Ansible on managed hosts.

Prerequisites

  • You prepared the control node. For more information, see Preparing a control node on RHEL 9.
  • You have SSH access from the control node.

    Important

    Direct SSH access as the root user is a security risk. To reduce this risk, you will create a local user on this node and configure a sudo policy when preparing a managed node. Ansible on the control node can then use the local user account to log in to the managed node and run playbooks as different users, such as root.

Procedure

  1. Create a user named ansible:

    [root@managed-node-01]# useradd ansible

    The control node later uses this user to establish an SSH connection to this host.

  2. Set a password for the ansible user:

    [root@managed-node-01]# passwd ansible
    Changing password for user ansible.
    New password: <password>
    Retype new password: <password>
    passwd: all authentication tokens updated successfully.

    You must enter this password when Ansible uses sudo to perform tasks as the root user.

  3. Install the ansible user’s SSH public key on the managed node:

    1. Log in to the control node as the ansible user, and copy the SSH public key to the managed node:

      [ansible@control-node]$ ssh-copy-id managed-node-01.example.com
      /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/ansible/.ssh/id_rsa.pub"
      The authenticity of host 'managed-node-01.example.com (192.0.2.100)' can't be established.
      ECDSA key fingerprint is SHA256:9bZ33GJNODK3zbNhybokN/6Mq7hu3vpBXDrCxe7NAvo.
    2. When prompted, connect by entering yes:

      Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
      /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
      /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
    3. When prompted, enter the password:

      ansible@managed-node-01.example.com's password: <password>
      
      Number of key(s) added: 1
      
      Now try logging into the machine, with:   "ssh 'managed-node-01.example.com'"
      and check to make sure that only the key(s) you wanted were added.
    4. Verify the SSH connection by remotely executing a command on the control node:

      [ansible@control-node]$ ssh managed-node-01.example.com whoami
      ansible
  4. Create a sudo configuration for the ansible user:

    1. Create and edit the /etc/sudoers.d/ansible file by using the visudo command:

      [root@managed-node-01]# visudo /etc/sudoers.d/ansible

      The benefit of using visudo over a normal editor is that this utility provides basic sanity checks and checks for parse errors before installing the file.

    2. Configure a sudoers policy in the /etc/sudoers.d/ansible file that meets your requirements, for example:

      • To grant permissions to the ansible user to run all commands as any user and group on this host after entering the ansible user’s password, use:

        ansible   ALL=(ALL) ALL
      • To grant permissions to the ansible user to run all commands as any user and group on this host without entering the ansible user’s password, use:

        ansible   ALL=(ALL) NOPASSWD: ALL

    Alternatively, configure a more fine-granular policy that matches your security requirements. For further details on sudoers policies, see the sudoers(5) manual page.

Verification

  1. Verify that you can execute commands from the control node on an all managed nodes:

    [ansible@control-node]$ ansible all -m ping
    BECOME password: <password>
    managed-node-01.example.com | SUCCESS => {
        	"ansible_facts": {
        	    "discovered_interpreter_python": "/usr/bin/python3"
        	},
        	"changed": false,
        	"ping": "pong"
    }
    ...

    The hard-coded all group dynamically contains all hosts listed in the inventory file.

  2. Verify that privilege escalation works correctly by running the whoami utility on a managed host by using the Ansible command module:

    [ansible@control-node]$ ansible managed-node-01.example.com -m command -a whoami
    BECOME password: <password>
    managed-node-01.example.com | CHANGED | rc=0 >>
    root

    If the command returns root, you configured sudo on the managed nodes correctly.

Additional resources

4.2. Introduction to the metrics system role

RHEL system roles is a collection of Ansible roles and modules that provide a consistent configuration interface to remotely manage multiple RHEL systems. The metrics system role configures performance analysis services for the local system and, optionally, includes a list of remote systems to be monitored by the local system. The metrics system role enables you to use pcp to monitor your systems performance without having to configure pcp separately, as the set-up and deployment of pcp is handled by the playbook.

Additional resources

  • /usr/share/ansible/roles/rhel-system-roles.metrics/README.md file
  • /usr/share/doc/rhel-system-roles/metrics/ directory

4.3. Using the metrics system role to monitor your local system with visualization

This procedure describes how to use the metrics RHEL system role to monitor your local system while simultaneously provisioning data visualization via Grafana.

Prerequisites

  • You have prepared the control node and the managed nodes
  • You are logged in to the control node as a user who can run playbooks on the managed nodes.
  • The account you use to connect to the managed nodes has sudo permissions on them.
  • localhost is configured in the inventory file on the control node:

    localhost ansible_connection=local

Procedure

  1. Create a playbook file, for example ~/playbook.yml, with the following content:

    ---
    - name: Manage metrics
      hosts: localhost
      roles:
        - rhel-system-roles.metrics
      vars:
        metrics_graph_service: yes
        metrics_manage_firewall: true
        metrics_manage_selinux: true

    Because the metrics_graph_service boolean is set to value="yes", Grafana is automatically installed and provisioned with pcp added as a data source. Because metrics_manage_firewall and metrics_manage_selinux are both set to true, the metrics role uses the firewall and selinux system roles to manage the ports used by the metrics role.

  2. Validate the playbook syntax:

    $ ansible-playbook --syntax-check ~/playbook.yml

    Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

  3. Run the playbook:

    $ ansible-playbook ~/playbook.yml

Verification

  • To view visualization of the metrics being collected on your machine, access the grafana web interface as described in Accessing the Grafana web UI.

Additional resources

  • /usr/share/ansible/roles/rhel-system-roles.metrics/README.md file
  • /usr/share/doc/rhel-system-roles/metrics/ directory

4.4. Using the metrics system role to set up a fleet of individual systems to monitor themselves

This procedure describes how to use the metrics system role to set up a fleet of machines to monitor themselves.

Prerequisites

Procedure

  1. Create a playbook file, for example ~/playbook.yml, with the following content:

    ---
    - name: Configure a fleet of machines to monitor themselves
      hosts: managed-node-01.example.com
      roles:
        - rhel-system-roles.metrics
      vars:
        metrics_retention_days: 0
        metrics_manage_firewall: true
        metrics_manage_selinux: true

    Because metrics_manage_firewall and metrics_manage_selinux are both set to true, the metrics role uses the firewall and selinux roles to manage the ports used by the metrics role.

  2. Validate the playbook syntax:

    $ ansible-playbook --syntax-check ~/playbook.yml

    Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

  3. Run the playbook:

    $ ansible-playbook ~/playbook.yml

Additional resources

  • /usr/share/ansible/roles/rhel-system-roles.metrics/README.md file
  • /usr/share/doc/rhel-system-roles/metrics/ directory

4.5. Using the metrics system role to monitor a fleet of machines centrally via your local machine

This procedure describes how to use the metrics system role to set up your local machine to centrally monitor a fleet of machines while also provisioning visualization of the data via grafana and querying of the data via redis.

Prerequisites

  • You have prepared the control node and the managed nodes
  • You are logged in to the control node as a user who can run playbooks on the managed nodes.
  • The account you use to connect to the managed nodes has sudo permissions on them.
  • localhost is configured in the inventory file on the control node:

    localhost ansible_connection=local

Procedure

  1. Create a playbook file, for example ~/playbook.yml, with the following content:

    - name: Set up your local machine to centrally monitor a fleet of machines
      hosts: localhost
      roles:
        - rhel-system-roles.metrics
      vars:
        metrics_graph_service: yes
        metrics_query_service: yes
        metrics_retention_days: 10
        metrics_monitored_hosts: ["database.example.com", "webserver.example.com"]
        metrics_manage_firewall: yes
        metrics_manage_selinux: yes

    Because the metrics_graph_service and metrics_query_service booleans are set to value="yes", grafana is automatically installed and provisioned with pcp added as a data source with the pcp data recording indexed into redis, allowing the pcp querying language to be used for complex querying of the data. Because metrics_manage_firewall and metrics_manage_selinux are both set to true, the metrics role uses the firewall and selinux roles to manage the ports used by the metrics role.

  2. Validate the playbook syntax:

    $ ansible-playbook --syntax-check ~/playbook.yml

    Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

  3. Run the playbook:

    $ ansible-playbook ~/playbook.yml

Verification

  • To view a graphical representation of the metrics being collected centrally by your machine and to query the data, access the grafana web interface as described in Accessing the Grafana web UI.

Additional resources

  • /usr/share/ansible/roles/rhel-system-roles.metrics/README.md file
  • /usr/share/doc/rhel-system-roles/metrics/ directory

4.6. Setting up authentication while monitoring a system by using the metrics system role

PCP supports the scram-sha-256 authentication mechanism through the Simple Authentication Security Layer (SASL) framework. The metrics RHEL system role automates the steps to setup authentication by using the scram-sha-256 authentication mechanism. This procedure describes how to setup authentication by using the metrics RHEL system role.

Prerequisites

Procedure

  1. Edit an existing playbook file, for example ~/playbook.yml, and add the authentication-related variables:

    ---
    - name: Set up authentication by using the scram-sha-256 authentication mechanism
      hosts: managed-node-01.example.com
      roles:
        - rhel-system-roles.metrics
      vars:
        metrics_retention_days: 0
        metrics_manage_firewall: true
        metrics_manage_selinux: true
        metrics_username: <username>
        metrics_password: <password>
  2. Validate the playbook syntax:

    $ ansible-playbook --syntax-check ~/playbook.yml

    Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

  3. Run the playbook:

    $ ansible-playbook ~/playbook.yml

Verification

  • Verify the sasl configuration:

    # pminfo -f -h "pcp://managed-node-01.example.com?username=<username>" disk.dev.read
    Password: <password>
    disk.dev.read
    inst [0 or "sda"] value 19540

Additional resources

  • /usr/share/ansible/roles/rhel-system-roles.metrics/README.md file
  • /usr/share/doc/rhel-system-roles/metrics/ directory

4.7. Using the metrics system role to configure and enable metrics collection for SQL Server

This procedure describes how to use the metrics RHEL system role to automate the configuration and enabling of metrics collection for Microsoft SQL Server via pcp on your local system.

Prerequisites

Procedure

  1. Create a playbook file, for example ~/playbook.yml, with the following content:

    ---
    - name: Configure and enable metrics collection for Microsoft SQL Server
      hosts: localhost
      roles:
        - rhel-system-roles.metrics
      vars:
        metrics_from_mssql: true
        metrics_manage_firewall: true
        metrics_manage_selinux: true

    Because metrics_manage_firewall and metrics_manage_selinux are both set to true, the metrics role uses the firewall and selinux roles to manage the ports used by the metrics role.

  2. Validate the playbook syntax:

    $ ansible-playbook --syntax-check ~/playbook.yml

    Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

  3. Run the playbook:

    $ ansible-playbook ~/playbook.yml

Verification

  • Use the pcp command to verify that SQL Server PMDA agent (mssql) is loaded and running:

    # pcp
    platform: Linux sqlserver.example.com 4.18.0-167.el8.x86_64 #1 SMP Sun Dec 15 01:24:23 UTC 2019 x86_64
     hardware: 2 cpus, 1 disk, 1 node, 2770MB RAM
     timezone: PDT+7
     services: pmcd pmproxy
         pmcd: Version 5.0.2-1, 12 agents, 4 clients
         pmda: root pmcd proc pmproxy xfs linux nfsclient mmv kvm mssql
               jbd2 dm
     pmlogger: primary logger: /var/log/pcp/pmlogger/sqlserver.example.com/20200326.16.31
         pmie: primary engine: /var/log/pcp/pmie/sqlserver.example.com/pmie.log

Additional resources

  • /usr/share/ansible/roles/rhel-system-roles.metrics/README.md file
  • /usr/share/doc/rhel-system-roles/metrics/ directory