Procedure

1. Edit /etc/tuned/cpu-partitioning-variables.conf file and add the following information:

   # Isolated CPUs with the kernel’s scheduler load balancing:
   isolated_cores=2-23
   # Isolated CPUs without the kernel’s scheduler load balancing:
   no_balance_cores=2,3

2. Set the cpu-partitioning TuneD profile:

   # tuned-adm profile cpu-partitioning

3. Reboot

   After rebooting, the system is tuned for low-latency, according to the isolation in the cpu-partitioning figure. The application can use taskset to pin the reader and writer threads to CPUs 2 and 3, and the remaining application threads on CPUs 4-23.

Procedure

1. Create the /etc/tuned/my_profile directory:

   # mkdir /etc/tuned/my_profile

2. Create a tuned.conf file in this directory, and add the following content:

   # vi /etc/tuned/my_profile/tuned.conf
   [main]
   summary=Customized tuning on top of cpu-partitioning
   include=cpu-partitioning
   [cpu]
   force_latency=cstate.id:0|1

3. Use the new profile:

   # tuned-adm profile my_profile

Procedure

1. Install the tuned package:

   # dnf install tuned

2. Enable and start the tuned service:

   # systemctl enable --now tuned

3. Optionally, install TuneD profiles for real-time systems:

   # dnf install tuned-profiles-realtime tuned-profiles-nfv

4. Verify that a TuneD profile is active and applied:

   $ tuned-adm active
   
   Current active profile: balanced

   $ tuned-adm verify
   
   Verfication succeeded, current system settings match the preset profile.
   See tuned log file ('/var/log/tuned/tuned.log') for details.

Procedure

1. Optionally, you can let TuneD recommend the most suitable profile for your system:

   # tuned-adm recommend
   
   balanced

2. Activate a profile:

   # tuned-adm profile selected-profile

   Alternatively, you can activate a combination of multiple profiles:

   # tuned-adm profile profile1 profile2

   Example 1.3. A virtual machine optimized for low power consumption

   The following example optimizes the system to run in a virtual machine with the best performance and concurrently tunes it for low power consumption, while the low power consumption is the priority:

   # tuned-adm profile virtual-guest powersave

3. View the current active TuneD profile on your system:

   # tuned-adm active
   
   Current active profile: selected-profile

4. Reboot the system:

   # reboot

Procedure

1. In the /etc/tuned/ directory, create a new directory named the same as the profile that you want to create:

   # mkdir /etc/tuned/my-profile

2. In the new directory, create a file named tuned.conf. Add a [main] section and plug-in definitions in it, according to your requirements.

   For example, see the configuration of the balanced profile:

   [main]
   summary=General non-specialized tuned profile
   
   [cpu]
   governor=conservative
   energy_perf_bias=normal
   
   [audio]
   timeout=10
   
   [video]
   radeon_powersave=dpm-balanced, auto
   
   [scsi_host]
   alpm=medium_power

3. To activate the profile, use:

   # tuned-adm profile my-profile

4. Verify that the TuneD profile is active and the system settings are applied:

   $ tuned-adm active
   
   Current active profile: my-profile

   $ tuned-adm verify
   
   Verfication succeeded, current system settings match the preset profile.
   See tuned log file ('/var/log/tuned/tuned.log') for details.

Procedure

1. In the /etc/tuned/ directory, create a new directory named the same as the profile that you want to create:

   # mkdir /etc/tuned/modified-profile

2. In the new directory, create a file named tuned.conf, and set the [main] section as follows:

   [main]
   include=parent-profile

   Replace parent-profile with the name of the profile you are modifying.

3. Include your profile modifications.

   Example 2.10. Lowering swappiness in the throughput-performance profile

   To use the settings from the throughput-performance profile and change the value of vm.swappiness to 5, instead of the default 10, use:

   [main]
   include=throughput-performance
   
   [sysctl]
   vm.swappiness=5

4. To activate the profile, use:

   # tuned-adm profile modified-profile

5. Verify that the TuneD profile is active and the system settings are applied:

   $ tuned-adm active
   
   Current active profile: my-profile

   $ tuned-adm verify
   
   Verfication succeeded, current system settings match the preset profile.
   See tuned log file ('/var/log/tuned/tuned.log') for details.

Procedure

1. Optional: Select an existing TuneD profile on which your profile will be based. For a list of available profiles, see TuneD profiles distributed with RHEL.

   To see which profile is currently active, use:

   $ tuned-adm active

2. Create a new directory to hold your TuneD profile:

   # mkdir /etc/tuned/my-profile

3. Find the system unique identifier of the selected block device:

   $ udevadm info --query=property --name=/dev/device | grep -E '(WWN|SERIAL)'
   
   ID_WWN=0x5002538d00000000_
   ID_SERIAL=Generic-_SD_MMC_20120501030900000-0:0
   ID_SERIAL_SHORT=20120501030900000

   Note

   The command in the this example will return all values identified as a World Wide Name (WWN) or serial number associated with the specified block device. Although it is preferred to use a WWN, the WWN is not always available for a given device and any values returned by the example command are acceptable to use as the device system unique ID.

4. Create the /etc/tuned/my-profile/tuned.conf configuration file. In the file, set the following options:

   1. Optional: Include an existing profile:

      [main]
      include=existing-profile

   2. Set the selected disk scheduler for the device that matches the WWN identifier:

      [disk]
      devices_udev_regex=IDNAME=device system unique id
      elevator=selected-scheduler

      Here:

      • Replace IDNAME with the name of the identifier being used (for example, ID_WWN).

      • Replace device system unique id with the value of the chosen identifier (for example, 0x5002538d00000000).

        To match multiple devices in the devices_udev_regex option, enclose the identifiers in parentheses and separate them with vertical bars:

        devices_udev_regex=(ID_WWN=0x5002538d00000000)|(ID_WWN=0x1234567800000000)

5. Enable your profile:

   # tuned-adm profile my-profile

Verification steps

1. Verify that the TuneD profile is active and applied:

   $ tuned-adm active
   
   Current active profile: my-profile

   $ tuned-adm verify
   
   Verification succeeded, current system settings match the preset profile.
   See tuned log file ('/var/log/tuned/tuned.log') for details.

2. Read the contents of the /sys/block/device/queue/scheduler file:

   # cat /sys/block/device/queue/scheduler
   
   [mq-deadline] kyber bfq none

   In the file name, replace device with the block device name, for example sdc.

   The active scheduler is listed in square brackets ([]).

Procedure

1. Install the rhel-system-roles package on the system that you want to use as a control node:

   # dnf install rhel-system-roles

2. Install the Ansible Core package:

   # dnf install ansible-core

Procedure

1. Create the required Ansible inventory containing the hosts and groups that you want to manage. Here is an example using a file called inventory.ini of a group of hosts called webservers:

   [webservers]
   host1
   host2
   host3

2. Create an Ansible playbook including the required role. The following example shows how to use roles through the roles: option for a playbook:

   The following example shows how to use roles through the roles: option for a given play:

   ---
   - hosts: webservers
     roles:
   
        - rhel-system-roles.network
        - rhel-system-roles.postfix

   Note

   Every role includes a README file, which documents how to use the role and supported parameter values. You can also find an example playbook for a particular role under the documentation directory of the role. Such documentation directory is provided by default with the rhel-system-roles package, and can be found in the following location:

   /usr/share/doc/rhel-system-roles/SUBSYSTEM/

   Replace SUBSYSTEM with the name of the required role, such as postfix, metrics, network, tlog, or ssh.

3. To execute the playbook on specific hosts, you must perform one of the following:

   • Edit the playbook to use hosts: host1[,host2,…​], or hosts: all, and execute the command:

     # ansible-playbook name.of.the.playbook

   • Edit the inventory to ensure that the hosts you want to use are defined in a group, and execute the command:

     # ansible-playbook -i name.of.the.inventory name.of.the.playbook

   • Specify all hosts when executing the ansible-playbook command:

     # ansible-playbook -i host1,host2,... name.of.the.playbook

     Important

     Be aware that the -i flag specifies the inventory of all hosts that are available. If you have multiple targeted hosts, but want to select a host against which you want to run the playbook, you can add a variable in the playbook to be able to select a host. For example:

     Ansible Playbook | example-playbook.yml:
     
     
     - hosts: "{{ target_host }}"
       roles:
          - rhel-system-roles.network
          - rhel-system-roles.postfix

     Playbook execution command:

     # ansible-playbook -i host1,..hostn -e target_host=host5 example-playbook.yml

Procedure

1. Configure localhost in the the /etc/ansible/hosts Ansible inventory by adding the following content to the inventory:

   localhost ansible_connection=local

2. Create an Ansible playbook with the following content:

   ---
   - hosts: localhost
     vars:
       metrics_graph_service: yes
     roles:
       - rhel-system-roles.metrics

3. Run the Ansible playbook:

   # ansible-playbook name_of_your_playbook.yml

   Note

   Since the metrics_graph_service boolean is set to value="yes", Grafana is automatically installed and provisioned with pcp added as a data source.

4. To view visualization of the metrics being collected on your machine, access the grafana web interface as described in Accessing the Grafana web UI.

Procedure

1. Add the name or IP of the machines you wish to monitor via the playbook to the /etc/ansible/hosts Ansible inventory file under an identifying group name enclosed in brackets:

   [remotes]
   webserver.example.com
   database.example.com

2. Create an Ansible playbook with the following content:

   ---
   - hosts: remotes
     vars:
       metrics_retention_days: 0
     roles:
       - rhel-system-roles.metrics

3. Run the Ansible playbook:

   # ansible-playbook name_of_your_playbook.yml -k

Procedure

1. Create an Ansible playbook with the following content:

   ---
   - hosts: localhost
     vars:
       metrics_graph_service: yes
       metrics_query_service: yes
       metrics_retention_days: 10
       metrics_monitored_hosts: ["database.example.com", "webserver.example.com"]
     roles:
       - rhel-system-roles.metrics

2. Run the Ansible playbook:

   # ansible-playbook name_of_your_playbook.yml

   Note

   Since the metrics_graph_service and metrics_query_service booleans are set to value="yes", grafana is automatically installed and provisioned with pcp added as a data source with the pcp data recording indexed into redis, allowing the pcp querying language to be used for complex querying of the data.

3. To view graphical representation of the metrics being collected centrally by your machine and to query the data, access the grafana web interface as described in Accessing the Grafana web UI.

Procedure

1. Include the following variables in the Ansible playbook you want to setup authentication for:

   ---
     vars:
       metrics_username: your_username
       metrics_password: your_password

2. Run the Ansible playbook:

   # ansible-playbook name_of_your_playbook.yml

Procedure

1. Configure localhost in the the /etc/ansible/hosts Ansible inventory by adding the following content to the inventory:

   localhost ansible_connection=local

2. Create an Ansible playbook that contains the following content:

   ---
   - hosts: localhost
     roles:
       - role: rhel-system-roles.metrics
         vars:
           metrics_from_mssql: yes

3. Run the Ansible playbook:

   # ansible-playbook name_of_your_playbook.yml

Procedure

1. Install the pcp package:

   # dnf install pcp

2. Enable and start the pmcd service on the host machine:

   # systemctl enable pmcd
   
   # systemctl start pmcd

Procedure

1. Update the pmlogger configuration:

   # pmlogconf -r /var/lib/pcp/config/pmlogger/config.default

2. Start the pmcd and pmlogger services:

   # systemctl start pmcd.service
   
   # systemctl start pmlogger.service

3. Execute the required operations to record the performance data.

4. Stop the pmcd and pmlogger services:

   # systemctl stop pmcd.service
   
   # systemctl stop pmlogger.service

5. Save the output and save it to a tar.gz file named based on the host name and the current date and time:

   # cd /var/log/pcp/pmlogger/
   
   # tar -czf $(hostname).$(date +%F-%Hh%M).pcp.tar.gz $(hostname)

   Extract this file and analyze the data using PCP tools.

Prerequisites

1. Install the pcp-pmda-redis package:

   # dnf install pcp-pmda-redis

2. Install the redis PMDA:

   # cd /var/lib/pcp/pmdas/redis && ./Install

Procedure

1. Create or modify the pmlogger configuration file:

   # pmlogconf -r /var/lib/pcp/config/pmlogger/config.default

2. Follow pmlogconf prompts to enable or disable groups of related performance metrics and to control the logging interval for each enabled group.

Procedure

1. Install the pcp-system-tools package:

   # dnf install pcp-system-tools

2. Configure an IP address for pmcd:

   # echo "-i 192.168.4.62" >>/etc/pcp/pmcd/pmcd.options

   Replace 192.168.4.62 with the IP address, the client should listen on.

   By default, pmcd is listening on the localhost.

3. Configure the firewall to add the public zone permanently:

   # firewall-cmd --permanent --zone=public --add-port=44321/tcp
   success
   
   # firewall-cmd --reload
   success

4. Set an SELinux boolean:

   # setsebool -P pcp_bind_all_unreserved_ports on

5. Enable the pmcd and pmlogger services:

   # systemctl enable pmcd pmlogger
   # systemctl restart pmcd pmlogger

Procedure

1. Install the pcp-system-tools package:

   # dnf install pcp-system-tools

2. Create the /etc/pcp/pmlogger/control.d/remote file with the following content:

   # DO NOT REMOVE OR EDIT THE FOLLOWING LINE
   $version=1.1
   
   192.168.4.13 n n PCP_ARCHIVE_DIR/rhel7u4a -r -T24h10m -c config.rhel7u4a
   192.168.4.14 n n PCP_ARCHIVE_DIR/rhel6u10a -r -T24h10m -c config.rhel6u10a
   192.168.4.62 n n PCP_ARCHIVE_DIR/rhel8u1a -r -T24h10m -c config.rhel8u1a
   192.168.4.69 n n PCP_ARCHIVE_DIR/rhel9u3a -r -T24h10m -c config.rhel9u3a

   Replace 192.168.4.13, 192.168.4.14, 192.168.4.62 and 192.168.4.69 with the client IP addresses.

3. Enable the pmcd and pmlogger services:

   # systemctl enable pmcd pmlogger
   # systemctl restart pmcd pmlogger

Procedure

1. Install the following packages:

   1. Install the pcp-system-tools:

      # dnf install pcp-system-tools

   2. Install the pmda-postfix package to monitor postfix:

      # dnf install pcp-pmda-postfix postfix

   3. Install the logging daemon:

      # dnf install rsyslog

   4. Install the mail client for testing:

      # dnf install mutt

2. Enable the postfix and rsyslog services:

   # systemctl enable postfix rsyslog
   # systemctl restart postfix rsyslog

3. Enable the SELinux boolean, so that pmda-postfix can access the required log files:

   # setsebool -P pcp_read_generic_logs=on

4. Install the PMDA:

   # cd /var/lib/pcp/pmdas/postfix/
   
   # ./Install
   
   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD ...
   Waiting for pmcd to terminate ...
   Starting pmcd ...
   Check postfix metrics have appeared ... 7 metrics and 58 values

Procedure

1. Launch the PCP Charts application from the command line:

   # pmchart

   Figure 7.1. PCP Charts application

   

   The pmtime server settings are located at the bottom. The start and pause button allows you to control:

   • The interval in which PCP polls the metric data
   • The date and time for the metrics of historical data
2. Click File and then New Chart to select metric from both the local machine and remote machines by specifying their host name or address. Advanced configuration options include the ability to manually set the axis values for the chart, and to manually choose the color of the plots.

3. Record the views created in the PCP Charts application:

   Following are the options to take images or record the views created in the PCP Charts application:

   • Click File and then Export to save an image of the current view.
   • Click Record and then Start to start a recording. Click Record and then Stop to stop the recording. After stopping the recording, the recorded metrics are archived to be viewed later.

4. Optional: In the PCP Charts application, the main configuration file, known as the view, allows the metadata associated with one or more charts to be saved. This metadata describes all chart aspects, including the metrics used and the chart columns. Save the custom view configuration by clicking File and then Save View, and load the view configuration later.

   The following example of the PCP Charts application view configuration file describes a stacking chart graph showing the total number of bytes read and written to the given XFS file system loop1:

   #kmchart
   version 1
   
   chart title "Filesystem Throughput /loop1" style stacking antialiasing off
       plot legend "Read rate"   metric xfs.read_bytes   instance  "loop1"
       plot legend "Write rate"  metric xfs.write_bytes  instance  "loop1"

Procedure

1. Install PCP:

   # dnf install pcp-zeroconf

2. Install packages required for the pyodbc driver:

   # dnf install python3-pyodbc

3. Install the mssql agent:

   1. Install the Microsoft SQL Server domain agent for PCP:

      # dnf install pcp-pmda-mssql

   2. Edit the /etc/pcp/mssql/mssql.conf file to configure the SQL server account’s username and password for the mssql agent. Ensure that the account you configure has access rights to performance data.

      username: user_name
      password: user_password

      Replace user_name with the SQL Server account and user_password with the SQL Server user password for this account.

4. Install the agent:

   # cd /var/lib/pcp/pmdas/mssql
   # ./Install
   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD ...
   Check mssql metrics have appeared ... 168 metrics and 598 values
   [...]

Procedure

1. Navigate to the xfs directory:

   # cd /var/lib/pcp/pmdas/xfs/

Procedure

1. Display the value of a metric:

   $ pminfo -f xfs.write
   
   xfs.write
       value 325262

2. Reset all the XFS metrics:

   # pmstore xfs.control.reset 1
   
   xfs.control.reset old value=0 new value=1

Procedure

1. Install the following packages:

   # dnf install grafana grafana-pcp

2. Restart and enable the following service:

   # systemctl restart grafana-server
   # systemctl enable grafana-server

3. Open the server’s firewall for network traffic to the Grafana service.

   # firewall-cmd --permanent --add-service=grafana
   success
   
   # firewall-cmd --reload
   success

Prerequisites

1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
2. The grafana-server is configured. For more information, see Setting up a grafana-server.

Procedure

1. On the client system, open a browser and access the grafana-server on port 3000, using http://192.0.2.0:3000 link.

   Replace 192.0.2.0 with your machine IP.

2. For the first login, enter admin in both the Email or username and Password field.

   Grafana prompts to set a New password to create a secured account. If you want to set it later, click Skip.

3. From the menu, hover over the       Configuration icon and then click Plugins.
4. In the Plugins tab, type performance co-pilot in the Search by name or type text box and then click Performance Co-Pilot (PCP) plugin.
5. In the Plugins / Performance Co-Pilot pane, click Enable.

6. Click Grafana       icon. The Grafana Home page is displayed.

   Figure 9.1. Home Dashboard

   
   Note

   The top corner of the screen has a similar       icon, but it controls the general Dashboard settings.

7. In the Grafana Home page, click Add your first data source to add PCP Redis, PCP bpftrace, and PCP Vector data sources. For more information on adding data source, see:

   • To add pcp redis data source, view default dashboard, create a panel, and an alert rule, see Creating panels and alert in PCP Redis data source.
   • To add pcp bpftrace data source and view the default dashboard, see Viewing the PCP bpftrace System Analysis dashboard.
   • To add pcp vector data source, view the default dashboard, and to view the vector checklist, see Viewing the PCP Vector Checklist.
8. Optional: From the menu, hover over the admin profile       icon to change the Preferences including Edit Profile, Change Password, or to Sign out.

Prerequisites

1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
2. The grafana-server is configured. For more information, see Setting up a grafana-server.

Procedure

1. Install the redis package:

   # dnf install redis

2. Start and enable the following services:

   # systemctl start pmproxy redis
   # systemctl enable pmproxy redis

3. Mail transfer agent, for example, sendmail or postfix is installed and configured.

4. Ensure that the allow_loading_unsigned_plugins parameter is set to PCP Redis database in the grafana.ini file:

   # vi /etc/grafana/grafana.ini
   
   allow_loading_unsigned_plugins = pcp-redis-datasource

5. Restart the grafana-server:

   # systemctl restart grafana-server

Prerequisites

1. The PCP Redis is configured. For more information, see Configuring PCP Redis.
2. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.

Procedure

1. Log into the Grafana web UI.
2. In the Grafana Home page, click Add your first data source.
3. In the Add data source pane, type redis in the Filter by name or type text box and then click PCP Redis.

4. In the Data Sources / PCP Redis pane, perform the following:

   1. Add http://localhost:44322 in the URL field and then click Save & Test.

   2. Click Dashboards tab → Import → PCP Redis: Host Overview to see a dashboard with an overview of any useful metrics.

      Figure 9.2. PCP Redis: Host Overview

      

5. Add a new panel:

   1. From the menu, hover over the       Create icon → Dashboard → Add new panel icon to add a panel.
   2. In the Query tab, select the PCP Redis from the query list instead of the selected default option and in the text field of A, enter metric, for example, kernel.all.load to visualize the kernel load graph.
   3. Optional: Add Panel title and Description, and update other options from the Settings.
   4. Click Save to apply changes and save the dashboard. Add Dashboard name.

   5. Click Apply to apply changes and go back to the dashboard.

      Figure 9.3. PCP Redis query panel

      

6. Create an alert rule:

   1. In the PCP Redis query panel, click       Alert and then click Create Alert.
   2. Edit the Name, Evaluate query, and For fields from the Rule, and specify the Conditions for your alert.

   3. Click Save to apply changes and save the dashboard. Click Apply to apply changes and go back to the dashboard.

      Figure 9.4. Creating alerts in the PCP Redis panel

      
   4. Optional: In the same panel, scroll down and click Delete icon to delete the created rule.

   5. Optional: From the menu, click       Alerting icon to view the created alert rules with different alert statuses, to edit the alert rule, or to pause the existing rule from the Alert Rules tab.

      To add a notification channel for the created alert rule to receive an alert notification from Grafana, see Adding notification channels for alerts.

Prerequisites

1. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.
2. An alert rule is created. For more information, see Creating panels and alert in PCP Redis data source.

3. Configure SMTP and add a valid sender’s email address in the grafana/grafana.ini file:

   # vi /etc/grafana/grafana.ini
   
   [smtp]
   enabled = true
   from_address = abc@gmail.com

   Replace abc@gmail.com by a valid email address.

Procedure

1. From the menu, hover over the       Alerting icon → click Notification channels → Add channel.

2. In the Add notification channel details pane, perform the following:

   1. Enter your name in the Name text box
   2. Select the communication Type, for example, Email and enter the email address. You can add multiple email addresses using the ; separator.
   3. Optional: Configure Optional Email settings and Notification settings.
3. Click Save.

4. Select a notification channel in the alert rule:

   1. From the menu, hover over the       Alerting icon and then click Alert rules.
   2. From the Alert Rules tab, click the created alert rule.
   3. On the Notifications tab, select your notification channel name from the Send to option, and then add an alert message.
   4. Click Apply.

Procedure

1. Install the sasl framework for the scram-sha-256 authentication mechanism:

   # dnf install cyrus-sasl-scram cyrus-sasl-lib

2. Specify the supported authentication mechanism and the user database path in the pmcd.conf file:

   # vi /etc/sasl2/pmcd.conf
   
   mech_list: scram-sha-256
   
   sasldb_path: /etc/pcp/passwd.db

3. Create a new user:

   # useradd -r metrics

   Replace metrics by your user name.

4. Add the created user in the user database:

   # saslpasswd2 -a pmcd metrics
   
   Password:
   Again (for verification):

   To add the created user, you are required to enter the metrics account password.

5. Set the permissions of the user database:

   # chown root:pcp /etc/pcp/passwd.db
   # chmod 640 /etc/pcp/passwd.db

6. Restart the pmcd service:

   # systemctl restart pmcd

Prerequisites

1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
2. The grafana-server is configured. For more information, see Setting up a grafana-server.
3. The scram-sha-256 authentication mechanism is configured. For more information, see Setting up authentication between PCP components.

Procedure

1. Install the pcp-pmda-bpftrace package:

   # dnf install pcp-pmda-bpftrace

2. Edit the bpftrace.conf file and add the user that you have created in the {setting-up-authentication-between-pcp-components}:

   # vi /var/lib/pcp/pmdas/bpftrace/bpftrace.conf
   
   [dynamic_scripts]
   enabled = true
   auth_enabled = true
   allowed_users = root,metrics

   Replace metrics by your user name.

3. Install bpftrace PMDA:

   # cd /var/lib/pcp/pmdas/bpftrace/
   # ./Install
   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD ...
   Check bpftrace metrics have appeared ... 7 metrics and 6 values

   The pmda-bpftrace is now installed, and can only be used after authenticating your user. For more information, see Viewing the PCP bpftrace System Analysis dashboard.

Prerequisites

1. The PCP bpftrace is installed. For more information, see Installing PCP bpftrace.
2. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.

Procedure

1. Log into the Grafana web UI.
2. In the Grafana Home page, click Add your first data source.
3. In the Add data source pane, type bpftrace in the Filter by name or type text box and then click PCP bpftrace.

4. In the Data Sources / PCP bpftrace pane, perform the following:

   1. Add http://localhost:44322 in the URL field.
   2. Toggle the Basic Auth option and add the created user credentials in the User and Password field.

   3. Click Save & Test.

      Figure 9.5. Adding PCP bpftrace in the data source

      

   4. Click Dashboards tab → Import → PCP bpftrace: System Analysis to see a dashboard with an overview of any useful metrics.

      Figure 9.6. PCP bpftrace: System Analysis

      

Prerequisites

1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
2. The grafana-server is configured. For more information, see Setting up a grafana-server.

Procedure

1. Install the pcp-pmda-bcc package:

   # dnf install pcp-pmda-bcc

2. Install the bcc PMDA:

   # cd /var/lib/pcp/pmdas/bcc
   # ./Install
   [Wed Apr  1 00:27:48] pmdabcc(22341) Info: Initializing, currently in 'notready' state.
   [Wed Apr  1 00:27:48] pmdabcc(22341) Info: Enabled modules:
   [Wed Apr  1 00:27:48] pmdabcc(22341) Info: ['biolatency', 'sysfork',
   [...]
   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD ...
   Check bcc metrics have appeared ... 1 warnings, 1 metrics and 0 values

Prerequisites

1. The PCP Vector is installed. For more information, see Installing PCP Vector.
2. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.

Procedure

1. Log into the Grafana web UI.
2. In the Grafana Home page, click Add your first data source.
3. In the Add data source pane, type vector in the Filter by name or type text box and then click PCP Vector.

4. In the Data Sources / PCP Vector pane, perform the following:

   1. Add http://localhost:44322 in the URL field and then click Save & Test.

   2. Click Dashboards tab → Import → PCP Vector: Host Overview to see a dashboard with an overview of any useful metrics.

      Figure 9.7. PCP Vector: Host Overview

      

5. From the menu, hover over the       Performance Co-Pilot plugin and then click PCP Vector Checklist.

   In the PCP checklist, click       help or       warning icon to view the related troubleshooting or reference links.

   Figure 9.8. Performance Co-Pilot / PCP Vector Checklist

   

Procedure

1. Log into the RHEL web console. For details, see Logging in to the web console.
2. Click Overview.

3. In the Performance Profile field, click the current performance profile.

   

4. In the Change Performance Profile dialog box, change the profile if necessary.

5. Click Change Profile.

   

Prerequisites

1. Make sure the web console is installed and accessible. For details, see Installing the web console.

2. Install the cockpit-pcp package, which enables collecting the performance metrics:

   # dnf install cockpit-pcp

Procedure

1. Log into the RHEL 9 web console. For details, see Logging in to the web console.

2. Click Overview.

   

3. Click View details and history to view the Performance Metrics.

   

   

Procedure

1. Optional: Select an existing TuneD profile on which your profile will be based. For a list of available profiles, see TuneD profiles distributed with RHEL.

   To see which profile is currently active, use:

   $ tuned-adm active

2. Create a new directory to hold your TuneD profile:

   # mkdir /etc/tuned/my-profile

3. Find the system unique identifier of the selected block device:

   $ udevadm info --query=property --name=/dev/device | grep -E '(WWN|SERIAL)'
   
   ID_WWN=0x5002538d00000000_
   ID_SERIAL=Generic-_SD_MMC_20120501030900000-0:0
   ID_SERIAL_SHORT=20120501030900000

   Note

   The command in the this example will return all values identified as a World Wide Name (WWN) or serial number associated with the specified block device. Although it is preferred to use a WWN, the WWN is not always available for a given device and any values returned by the example command are acceptable to use as the device system unique ID.

4. Create the /etc/tuned/my-profile/tuned.conf configuration file. In the file, set the following options:

   1. Optional: Include an existing profile:

      [main]
      include=existing-profile

   2. Set the selected disk scheduler for the device that matches the WWN identifier:

      [disk]
      devices_udev_regex=IDNAME=device system unique id
      elevator=selected-scheduler

      Here:

      • Replace IDNAME with the name of the identifier being used (for example, ID_WWN).

      • Replace device system unique id with the value of the chosen identifier (for example, 0x5002538d00000000).

        To match multiple devices in the devices_udev_regex option, enclose the identifiers in parentheses and separate them with vertical bars:

        devices_udev_regex=(ID_WWN=0x5002538d00000000)|(ID_WWN=0x1234567800000000)

5. Enable your profile:

   # tuned-adm profile my-profile

Verification steps

1. Verify that the TuneD profile is active and applied:

   $ tuned-adm active
   
   Current active profile: my-profile

   $ tuned-adm verify
   
   Verification succeeded, current system settings match the preset profile.
   See tuned log file ('/var/log/tuned/tuned.log') for details.

2. Read the contents of the /sys/block/device/queue/scheduler file:

   # cat /sys/block/device/queue/scheduler
   
   [mq-deadline] kyber bfq none

   In the file name, replace device with the block device name, for example sdc.

   The active scheduler is listed in square brackets ([]).

Procedure

1. Find the system unique identifier of the block device:

   $ udevadm info --name=/dev/device | grep -E '(WWN|SERIAL)'
   E: ID_WWN=0x5002538d00000000
   E: ID_SERIAL=Generic-_SD_MMC_20120501030900000-0:0
   E: ID_SERIAL_SHORT=20120501030900000

   Note

   The command in the this example will return all values identified as a World Wide Name (WWN) or serial number associated with the specified block device. Although it is preferred to use a WWN, the WWN is not always available for a given device and any values returned by the example command are acceptable to use as the device system unique ID.

2. Configure the udev rule. Create the /etc/udev/rules.d/99-scheduler.rules file with the following content:

   ACTION=="add|change", SUBSYSTEM=="block", ENV{IDNAME}=="device system unique id", ATTR{queue/scheduler}="selected-scheduler"

   Here:

   • Replace IDNAME with the name of the identifier being used (for example, ID_WWN).
   • Replace device system unique id with the value of the chosen identifier (for example, 0x5002538d00000000).

3. Reload udev rules:

   # udevadm control --reload-rules

4. Apply the scheduler configuration:

   # udevadm trigger --type=devices --action=change

Procedure

1. Remove the server max protocol parameter from the [global] section in the /etc/samba/smb.conf file.

2. Reload the Samba configuration

   # smbcontrol all reload-config

Procedure

1. Rename all files on the share to lowercase.

   Note

   Using the settings in this procedure, files with names other than in lowercase will no longer be displayed.

2. Set the following parameters in the share’s section:

   case sensitive = true
   default case = lower
   preserve case = no
   short preserve case = no

   For details about the parameters, see their descriptions in the smb.conf(5) man page.

3. Verify the /etc/samba/smb.conf file:

   # testparm

4. Reload the Samba configuration:

   # smbcontrol all reload-config

1. List the available tuned profiles.

   # tuned-adm list
   
   Available profiles:
   - balanced             - General non-specialized tuned profile
   - desktop              - Optimize for the desktop use-case
   [...]
   - virtual-guest        - Optimize for running inside a virtual guest
   - virtual-host         - Optimize for running KVM guests
   Current active profile: balanced

2. Optional: Create a new tuned profile or edit an existing tuned profile.

   For more information, see Customizing tuned profiles.

3. Activate a tuned profile.

   # tuned-adm profile selected-profile

   • To optimize a virtualization host, use the virtual-host profile.

     # tuned-adm profile virtual-host

   • On a RHEL guest operating system, use the virtual-guest profile.

     # tuned-adm profile virtual-guest

1. Adjust how many host CPUs are assigned to the VM. You can do this using the CLI or the web console.

2. Ensure that the vCPU model is aligned with the CPU model of the host. For example, to set the testguest1 VM to use the CPU model of the host:

   # virt-xml testguest1 --edit --cpu host-model

3. Manage kernel same-page merging (KSM).

4. If your host machine uses Non-Uniform Memory Access (NUMA), you can also configure NUMA for its VMs. This maps the host’s CPU and memory processes onto the CPU and memory processes of the VM as closely as possible. In effect, NUMA tuning provides the vCPU with a more streamlined access to the system memory allocated to the VM, which can improve the vCPU processing effectiveness.

   For details, see Configuring NUMA in a virtual machine and Sample vCPU performance tuning scenario.

1. On the host, install the perf package:

   # dnf install perf

2. Use one of the perf kvm stat commands to display perf statistics for your virtualization host:

   • For real-time monitoring of your hypervisor, use the perf kvm stat live command.
   • To log the perf data of your hypervisor over a period of time, activate the logging using the perf kvm stat record command. After the command is canceled or interrupted, the data is saved in the perf.data.guest file, which can be analyzed using the perf kvm stat report command.

3. Analyze the perf output for types of VM-EXIT events and their distribution. For example, the PAUSE_INSTRUCTION events should be infrequent, but in the following output, the high occurrence of this event suggests that the host CPUs are not handling the running vCPUs well. In such a scenario, consider shutting down some of your active VMs, removing vCPUs from these VMs, or tuning the performance of the vCPUs.

   # perf kvm stat report
   
   Analyze events for all VMs, all VCPUs:
   
   
                VM-EXIT    Samples  Samples%     Time%    Min Time    Max Time         Avg time
   
     EXTERNAL_INTERRUPT     365634    31.59%    18.04%      0.42us  58780.59us    204.08us ( +-   0.99% )
              MSR_WRITE     293428    25.35%     0.13%      0.59us  17873.02us      1.80us ( +-   4.63% )
       PREEMPTION_TIMER     276162    23.86%     0.23%      0.51us  21396.03us      3.38us ( +-   5.19% )
      PAUSE_INSTRUCTION     189375    16.36%    11.75%      0.72us  29655.25us    256.77us ( +-   0.70% )
                    HLT      20440     1.77%    69.83%      0.62us  79319.41us  14134.56us ( +-   0.79% )
                 VMCALL      12426     1.07%     0.03%      1.02us   5416.25us      8.77us ( +-   7.36% )
          EXCEPTION_NMI         27     0.00%     0.00%      0.69us      1.34us      0.98us ( +-   3.50% )
          EPT_MISCONFIG          5     0.00%     0.00%      5.15us     10.85us      7.88us ( +-  11.67% )
   
   Total Samples:1157497, Total events handled time:413728274.66us.

   Other event types that can signal problems in the output of perf kvm stat include:

   • INSN_EMULATION - suggests suboptimal VM I/O configuration.