Chapter 4. Monitoring performance using RHEL System Roles
As a system administrator, you can use the metrics
RHEL System Role to monitor the performance of a system.
4.1. Preparing a control node and managed nodes to use RHEL System Roles
Before you can use individual RHEL System Roles to manage services and settings, you must prepare the control node and managed nodes.
4.1.1. Preparing a control node on RHEL 9
Before using RHEL System Roles, you must configure a control node. This system then configures the managed hosts from the inventory according to the playbooks.
Prerequisites
- RHEL 8.6 or later is installed. For more information about installing RHEL, see Performing a standard RHEL 9 installation.
- The system is registered to the Customer Portal.
-
A
Red Hat Enterprise Linux Server
subscription is attached to the system. -
If available in your Customer Portal account, an
Ansible Automation Platform
subscription is attached to the system.
Procedure
Install the
rhel-system-roles
package:[root@control-node]# dnf install rhel-system-roles
This command installs the
ansible-core
package as a dependency.NoteIn RHEL 8.5 and earlier versions, Ansible packages were provided through Ansible Engine instead of Ansible Core, and with a different level of support. Do not use Ansible Engine because the packages might not be compatible with Ansible automation content in RHEL 8.6 and later. For more information, see Scope of support for the Ansible Core package included in the RHEL 9 and RHEL 8.6 and later AppStream repositories.
Create a user named
ansible
to manage and run playbooks:[root@control-node]# useradd ansible
Switch to the newly created
ansible
user:[root@control-node]# su - ansible
Perform the rest of the procedure as this user.
Create an SSH public and private key:
[ansible@control-node]$ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/home/ansible/.ssh/id_rsa): Enter passphrase (empty for no passphrase): <password> Enter same passphrase again: <password> ...
Use the suggested default location for the key file.
- Optional: To prevent Ansible from prompting you for the SSH key password each time you establish a connection, configure an SSH agent.
Create the
~/.ansible.cfg
file with the following content:[defaults] inventory = /home/ansible/inventory remote_user = ansible [privilege_escalation] become = True become_method = sudo become_user = root become_ask_pass = True
NoteSettings in the
~/.ansible.cfg
file have a higher priority and override settings from the global/etc/ansible/ansible.cfg
file.With these settings, Ansible performs the following actions:
- Manages hosts in the specified inventory file.
-
Uses the account set in the
remote_user
parameter when it establishes SSH connections to managed nodes. -
Uses the
sudo
utility to execute tasks on managed nodes as theroot
user. - Prompts for the root password of the remote user every time you apply a playbook. This is recommended for security reasons.
Create an
~/inventory
file in INI or YAML format that lists the hostnames of managed hosts. You can also define groups of hosts in the inventory file. For example, the following is an inventory file in the INI format with three hosts and one host group namedUS
:managed-node-01.example.com [US] managed-node-02.example.com ansible_host=192.0.2.100 managed-node-03.example.com
Note that the control node must be able to resolve the hostnames. If the DNS server cannot resolve certain hostnames, add the
ansible_host
parameter next to the host entry to specify its IP address.
Next steps
- Prepare the managed nodes. For more information, see Preparing a managed node.
Additional resources
- Scope of support for the Ansible Core package included in the RHEL 9 and RHEL 8.6 and later AppStream repositories
- How to register and subscribe a system to the Red Hat Customer Portal using subscription-manager
-
The
ssh-keygen(1)
man page - Connecting to remote machines with SSH keys using ssh-agent
- Ansible configuration settings
- How to build your inventory
4.1.2. Preparing a managed node
Managed nodes are the systems listed in the inventory and which will be configured by the control node according to the playbook. You do not have to install Ansible on managed hosts.
Prerequisites
- You prepared the control node. For more information, see Preparing a control node on RHEL 9.
You have SSH access from the control node.
ImportantDirect SSH access as the
root
user is a security risk. To reduce this risk, you will create a local user on this node and configure asudo
policy when preparing a managed node. Ansible on the control node can then use the local user account to log in to the managed node and run playbooks as different users, such asroot
.
Procedure
Create a user named
ansible
:[root@managed-node-01]# useradd ansible
The control node later uses this user to establish an SSH connection to this host.
Set a password for the
ansible
user:[root@managed-node-01]# passwd ansible Changing password for user ansible. New password: <password> Retype new password: <password> passwd: all authentication tokens updated successfully.
You must enter this password when Ansible uses
sudo
to perform tasks as theroot
user.Install the
ansible
user’s SSH public key on the managed node:Log in to the control node as the
ansible
user, and copy the SSH public key to the managed node:[ansible@control-node]$ ssh-copy-id managed-node-01.example.com /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/ansible/.ssh/id_rsa.pub" The authenticity of host 'managed-node-01.example.com (192.0.2.100)' can't be established. ECDSA key fingerprint is SHA256:9bZ33GJNODK3zbNhybokN/6Mq7hu3vpBXDrCxe7NAvo.
When prompted, connect by entering
yes
:Are you sure you want to continue connecting (yes/no/[fingerprint])? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
When prompted, enter the password:
ansible@managed-node-01.example.com's password: <password> Number of key(s) added: 1 Now try logging into the machine, with: "ssh '<managed-node-01.example.com>'" and check to make sure that only the key(s) you wanted were added.
Verify the SSH connection by remotely executing a command on the control node:
[ansible@control-node]$ ssh <managed-node-01.example.com> whoami ansible
Create a
sudo
configuration for theansible
user:Create and edit the
/etc/sudoers.d/ansible
file by using thevisudo
command:[root@managed-node-01]# visudo /etc/sudoers.d/ansible
The benefit of using
visudo
over a normal editor is that this utility provides basic sanity checks and checks for parse errors before installing the file.Configure a
sudoers
policy in the/etc/sudoers.d/ansible
file that meets your requirements, for example:To grant permissions to the
ansible
user to run all commands as any user and group on this host after entering theansible
user’s password, use:ansible ALL=(ALL) ALL
To grant permissions to the
ansible
user to run all commands as any user and group on this host without entering theansible
user’s password, use:ansible ALL=(ALL) NOPASSWD: ALL
Alternatively, configure a more fine-granular policy that matches your security requirements. For further details on
sudoers
policies, see thesudoers(5)
man page.
Verification
Verify that you can execute commands from the control node on an all managed nodes:
[ansible@control-node]$ ansible all -m ping BECOME password: <password> managed-node-01.example.com | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python3" }, "changed": false, "ping": "pong" } ...
The hard-coded all group dynamically contains all hosts listed in the inventory file.
Verify that privilege escalation works correctly by running the
whoami
utility on a managed host by using the Ansiblecommand
module:[ansible@control-node]$ ansible managed-node-01.example.com -m command -a whoami BECOME password: <password> managed-node-01.example.com | CHANGED | rc=0 >> root
If the command returns root, you configured
sudo
on the managed nodes correctly.
Additional resources
- Preparing a control node on RHEL 9.
-
The
sudoers(5)
man page
4.2. Introduction to the metrics
System Role
RHEL System Roles is a collection of Ansible roles and modules that provide a consistent configuration interface to remotely manage multiple RHEL systems. The metrics
System Role configures performance analysis services for the local system and, optionally, includes a list of remote systems to be monitored by the local system. The metrics
System Role enables you to use pcp
to monitor your systems performance without having to configure pcp
separately, as the set-up and deployment of pcp
is handled by the playbook.
Table 4.1. metrics
system role variables
Role variable | Description | Example usage |
---|---|---|
metrics_monitored_hosts |
List of remote hosts to be analyzed by the target host. These hosts will have metrics recorded on the target host, so ensure enough disk space exists below |
|
metrics_retention_days | Configures the number of days for performance data retention before deletion. |
|
metrics_graph_service |
A boolean flag that enables the host to be set up with services for performance data visualization via |
|
metrics_query_service |
A boolean flag that enables the host to be set up with time series query services for querying recorded |
|
metrics_provider |
Specifies which metrics collector to use to provide metrics. Currently, |
|
metrics_manage_firewall |
Uses the |
|
metrics_manage_selinux |
Uses the |
|
For details about the parameters used in metrics_connections
and additional information about the metrics
System Role, see the /usr/share/ansible/roles/rhel-system-roles.metrics/README.md
file.
4.3. Using the metrics
System Role to monitor your local system with visualization
This procedure describes how to use the metrics
RHEL System Role to monitor your local system while simultaneously provisioning data visualization via Grafana
.
Prerequisites
- The Ansible Core package is installed on the control machine.
-
You have the
rhel-system-roles
package installed on the machine you want to monitor.
Procedure
Configure
localhost
in the/etc/ansible/hosts
Ansible inventory by adding the following content to the inventory:localhost ansible_connection=local
Create an Ansible playbook with the following content:
--- - name: Manage metrics hosts: localhost vars: metrics_graph_service: yes metrics_manage_firewall: true metrics_manage_selinux: true roles: - rhel-system-roles.metrics
Run the Ansible playbook:
# ansible-playbook name_of_your_playbook.yml
NoteBecause the
metrics_graph_service
boolean is set tovalue="yes"
,Grafana
is automatically installed and provisioned withpcp
added as a data source. Becausemetrics_manage_firewall
andmetrics_manage_selinux
are both set totrue
, the metrics role uses thefirewall
andselinux
system roles to manage the ports used by the metrics role.-
To view visualization of the metrics being collected on your machine, access the
grafana
web interface as described in Accessing the Grafana web UI.
4.4. Using the metrics
System Role to set up a fleet of individual systems to monitor themselves
This procedure describes how to use the metrics
System Role to set up a fleet of machines to monitor themselves.
Prerequisites
- The Ansible Core package is installed on the control machine.
-
You have the
rhel-system-roles
package installed on the machine you want to use to run the playbook. - You have the SSH connection established.
Procedure
Add the name or IP address of the machines you want to monitor via the playbook to the
/etc/ansible/hosts
Ansible inventory file under an identifying group name enclosed in brackets:[remotes] webserver.example.com database.example.com
Create an Ansible playbook with the following content:
--- - hosts: remotes vars: metrics_retention_days: 0 metrics_manage_firewall: true metrics_manage_selinux: true roles: - rhel-system-roles.metrics
NoteBecause
metrics_manage_firewall
andmetrics_manage_selinux
are both set totrue
, the metrics role uses thefirewall
andselinux
roles to manage the ports used by themetrics
role.Run the Ansible playbook:
# ansible-playbook name_of_your_playbook.yml -k
Where the
-k
prompt for password to connect to remote system.
4.5. Using the metrics
System Role to monitor a fleet of machines centrally via your local machine
This procedure describes how to use the metrics
System Role to set up your local machine to centrally monitor a fleet of machines while also provisioning visualization of the data via grafana
and querying of the data via redis
.
Prerequisites
- The Ansible Core package is installed on the control machine.
-
You have the
rhel-system-roles
package installed on the machine you want to use to run the playbook.
Procedure
Create an Ansible playbook with the following content:
--- - hosts: localhost vars: metrics_graph_service: yes metrics_query_service: yes metrics_retention_days: 10 metrics_monitored_hosts: ["database.example.com", "webserver.example.com"] metrics_manage_firewall: yes metrics_manage_selinux: yes roles: - rhel-system-roles.metrics
Run the Ansible playbook:
# ansible-playbook name_of_your_playbook.yml
NoteBecause the
metrics_graph_service
andmetrics_query_service
booleans are set tovalue="yes"
,grafana
is automatically installed and provisioned withpcp
added as a data source with thepcp
data recording indexed intoredis
, allowing thepcp
querying language to be used for complex querying of the data. Becausemetrics_manage_firewall
andmetrics_manage_selinux
are both set totrue
, themetrics
role uses thefirewall
andselinux
roles to manage the ports used by themetrics
role.-
To view a graphical representation of the metrics being collected centrally by your machine and to query the data, access the
grafana
web interface as described in Accessing the Grafana web UI.
4.6. Setting up authentication while monitoring a system using the metrics
System Role
PCP supports the scram-sha-256
authentication mechanism through the Simple Authentication Security Layer (SASL) framework. The metrics
RHEL System Role automates the steps to setup authentication using the scram-sha-256
authentication mechanism. This procedure describes how to setup authentication using the metrics
RHEL System Role.
Prerequisites
- The Ansible Core package is installed on the control machine.
-
You have the
rhel-system-roles
package installed on the machine you want to use to run the playbook.
Procedure
Include the following variables in the Ansible playbook you want to setup authentication for:
--- vars: metrics_username: your_username metrics_password: your_password metrics_manage_firewall: true metrics_manage_selinux: true
NoteBecause
metrics_manage_firewall
andmetrics_manage_selinux
are both set totrue
, themetrics
role uses thefirewall
andselinux
roles to manage the ports used by themetrics
role.Run the Ansible playbook:
# ansible-playbook name_of_your_playbook.yml
Verification steps
Verify the
sasl
configuration:# pminfo -f -h "pcp://ip_adress?username=your_username" disk.dev.read Password: disk.dev.read inst [0 or "sda"] value 19540
ip_adress should be replaced by the IP address of the host.
4.7. Using the metrics
System Role to configure and enable metrics collection for SQL Server
This procedure describes how to use the metrics
RHEL System Role to automate the configuration and enabling of metrics collection for Microsoft SQL Server via pcp
on your local system.
Prerequisites
- The Ansible Core package is installed on the control machine.
-
You have the
rhel-system-roles
package installed on the machine you want to monitor. - You have installed Microsoft SQL Server for Red Hat Enterprise Linux and established a 'trusted' connection to an SQL server. See Install SQL Server and create a database on Red Hat.
- You have installed the Microsoft ODBC driver for SQL Server for Red Hat Enterprise Linux. See Red Hat Enterprise Server and Oracle Linux.
Procedure
Configure
localhost
in the/etc/ansible/hosts
Ansible inventory by adding the following content to the inventory:localhost ansible_connection=local
Create an Ansible playbook that contains the following content:
--- - hosts: localhost vars: metrics_from_mssql: true metrics_manage_firewall: true metrics_manage_selinux: true roles: - role: rhel-system-roles.metrics
NoteBecause
metrics_manage_firewall
andmetrics_manage_selinux
are both set totrue
, themetrics
role uses thefirewall
andselinux
roles to manage the ports used by themetrics
role.Run the Ansible playbook:
# ansible-playbook name_of_your_playbook.yml
Verification steps
Use the
pcp
command to verify that SQL Server PMDA agent (mssql) is loaded and running:# pcp platform: Linux rhel82-2.local 4.18.0-167.el8.x86_64 #1 SMP Sun Dec 15 01:24:23 UTC 2019 x86_64 hardware: 2 cpus, 1 disk, 1 node, 2770MB RAM timezone: PDT+7 services: pmcd pmproxy pmcd: Version 5.0.2-1, 12 agents, 4 clients pmda: root pmcd proc pmproxy xfs linux nfsclient mmv kvm mssql jbd2 dm pmlogger: primary logger: /var/log/pcp/pmlogger/rhel82-2.local/20200326.16.31 pmie: primary engine: /var/log/pcp/pmie/rhel82-2.local/pmie.log