Red Hat Training

A Red Hat training course is available for Red Hat Enterprise Linux

5.2. XFS File System Performance Analysis with Performance Co-Pilot

This section describes PCP XFS performance metrics and how to use them. Once started, the Performance Metric Collector Daemon (PMCD) begins collecting performance data from the installed Performance Metric Domain Agents (PMDAs). PMDAs can be individually loaded or unloaded on the system and are controlled by the PMCD on the same host. The XFS PMDA, which is part of the default PCP installation, is used to gather performance metric data of XFS file systems in PCP.
For a list of system services and tools that are distributed with PCP, see Table A.1, “System Services Distributed with Performance Co-Pilot in Red Hat Enterprise Linux 7” and Table A.2, “Tools Distributed with Performance Co-Pilot in Red Hat Enterprise Linux 7”.

5.2.1. Installing XFS PMDA to Gather XFS Data with PCP

The XFS PMDA ships as part of the pcp package and is enabled by default on installation. To install PCP, enter: 
# yum install pcp
To enable and start the PMDA service on the host machine after the pcp and pcp-gui packages are installed, use the following commands: 
# systemctl enable pmcd.service
# systemctl start pmcd.service
To query the PCP environment to verify that the PMCD process is running on the host and that the XFS PMDA is listed as enabled in the configuration, enter: 
# pcp

Performance Co-Pilot configuration on workstation:

platform: Linux workstation 3.10.0-123.20.1.el7.x86_64 #1 SMP Thu Jan
29 18:05:33 UTC 2015 x86_64
hardware: 2 cpus, 2 disks, 1 node, 2048MB RAM
timezone: BST-1
services pmcd
pmcd: Version 3.10.6-1, 7 agents
pmda: root pmcd proc xfs linux mmv jbd2

Installing XFS PMDA Manually

If the XFS PMDA is not listed in PCP configuration readout, install the PMDA agent manually. The PMDA installation script prompts you to specify the PMDA role: collector, monitor, or both.
  • The collector role allows the collection of performance metrics on the current system
  • The monitor role allows the system to monitor local systems, remote systems, or both.
The default option is both collector and monitor, which allows the XFS PMDA to operate correctly in most scenarios.
To install XFS PMDA manually, change to the xfs directory: 
# cd /var/lib/pcp/pmdas/xfs/
In the xfs directory, enter: 
xfs]# ./Install

You will need to choose an appropriate configuration for install of
the “xfs” Performance Metrics Domain Agent (PMDA).

  collector     collect performance statistics on this system
  monitor       allow this system to monitor local and/or remote systems
  both          collector and monitor configuration for this system

Please enter c(ollector) or m(onitor) or (both) [b]
Updating the Performance Metrics Name Space (PMNS) ...
Terminate PMDA if already installed ...
Updating the PMCD control file, and notifying PMCD ...
Waiting for pmcd to terminate ...
Starting pmcd ...
Check xfs metrics have appeared ... 149 metrics and 149 values

5.2.2. Configuring and Examining XFS Performance Metrics

Examining Metrics with pminfo

With PCP installed and the XFS PMDA enabled, instructions are available in Section 5.2.1, “Installing XFS PMDA to Gather XFS Data with PCP”, the easiest way to start looking at the performance metrics available for PCP and XFS is to use the pminfo tool, which displays information about available performance metrics. The command displays a list of all available metrics provided by the XFS PMDA.
To display a list of all available metrics provided by the XFS PMDA: 
# pminfo xfs
Use the following options to display information on selected metrics:
-t metric
Displays one-line help information describing the selected metric.
-T metric
Displays more verbose help text describing the selected metric.
-f metric
Displays the current reading of the performance value that corresponds to the metric.
You can use the -t, -T, and -f options with a group of metrics or an individual metric. Most metric data is provided for each mounted XFS file system on the system at time of probing.
There are different groups of XFS metrics, which are arranged so that each different group is a new leaf node from the root XFS metric, using a dot (.) as a separator. The leaf node semantics (dots) applies to all PCP metrics. For an overview of the types of metrics that are available in each of the groups, see Table A.3, “PCP Metric Groups for XFS”.
Additionally, the XFS documentation contains a section on monitoring XFS file systems: Chapter 13. XFS Monitoring.

Example 5.1. Using the pminfo Tool to Examine XFS Read and Write Metrics

To display one-line help information describing the xfs.write_bytes metric: 
# pminfo -t xfs.write_bytes

xfs.write_bytes [number of bytes written in XFS file system write operations]
To display more verbose help text describing the xfs.read_bytes metric: 
# pminfo -T xfs.read_bytes

xfs.read_bytes
Help:
This is the number of bytes read via read(2) system calls to files in
XFS file systems. It can be used in conjunction with the read_calls
count to calculate the average size of the read operations to file in
XFS file systems.
To obtain the current reading of the performance value that corresponds to the xfs.read_bytes metric: 
# pminfo -f xfs.read_bytes

xfs.read_bytes
    value 4891346238

Configuring Metrics with pmstore

With PCP, you can modify the values of certain metrics, especially if the metric acts as a control variable, for example the xfs.control.reset metric. To modify a metric value, use the pmstore tool.

Example 5.2. Using pmstore to Reset the xfs.control.reset Metric

This example shows how to use pmstore with the xfs.control.reset metric to reset the recorded counter values for the XFS PMDA back to zero. 
$ pminfo -f xfs.write

xfs.write
    value 325262

# pmstore xfs.control.reset 1

xfs.control.reset old value=0 new value=1

$ pminfo -f xfs.write

xfs.write
    value 0

5.2.3. Examining XFS Metrics Available per File System

Starting with Red Hat Enterprise Linux 7.3, PCP enables XFS PMDA to allow the reporting of certain XFS metrics per each of the mounted XFS file systems. This makes it easier to pinpoint specific mounted file system issues and evaluate performance. For an overview of the types of metrics available per file system in each of the groups, see Table A.4, “PCP Metric Groups for XFS per Device”.

Example 5.3. Obtaining per-Device XFS Metrics with pminfo

The pminfo command provides per-device XFS metrics that give instance values for each mounted XFS file system. 
# pminfo -f -t xfs.perdev.read xfs.perdev.write

xfs.perdev.read [number of XFS file system read operations]
inst [0 or "loop1"] value 0
inst [0 or "loop2"] value 0

xfs.perdev.write [number of XFS file system write operations]
inst [0 or "loop1"] value 86
inst [0 or "loop2"] value 0

5.2.4. Logging Performance Data with pmlogger

PCP allows you to log performance metric values that can be replayed later and used for a retrospective performance analysis. Use the pmlogger tool to create archived logs of selected metrics on the system.
With pmlogger, you can specify which metrics are recorded on the system and how often. The default pmlogger configuration file is /var/lib/pcp/config/pmlogger/config.default. The configuration file specifies which metrics are logged by the primary logging instance.
To log metric values on the local machine with pmlogger, start a primary logging instance: 
# systemctl start pmlogger.service
# systemctl enable pmlogger.service
When pmlogger is enabled and a default configuration file is set, a pmlogger line is included in the PCP configuration: 
# pcp

Performance Co-Pilot configuration on workstation:

platform: Linux workstation 3.10.0-123.20.1.el7.x86_64 #1 SMP Thu Jan
[...]
pmlogger: primary logger:/var/log/pcp/pmlogger/workstation/20160820.10.15

Modifying the pmlogger Configuration File with pmlogconf

When the pmlogger service is running, PCP logs a default set of metrics on the host. You can use the pmlogconf utility to check the default configuration, and enable XFS logging groups as needed. Important XFS groups to enable include the XFS information, XFS data, and log I/O traffic groups.
Follow pmlogconf prompts to enable or disable groups of related performance metrics, and to control the logging interval for each enabled group. Group selection is made by pressing y (yes) or n (no) in response to the prompt. To create or modify the generic PCP archive logger configuration file with pmlogconf, enter: 
# pmlogconf -r /var/lib/pcp/config/pmlogger/config.default

Modifying the pmlogger Configuration File Manually

You can edit the pmlogger configuration file manually and add specific metrics with given intervals to create a tailored logging configuration.

Example 5.4. The pmlogger Configuration File with XFS Metrics

The following example shows an extract of the pmlogger config.default file with some specific XFS metrics added. 
# It is safe to make additions from here on ...
#

log mandatory on every 5 seconds {
    xfs.write
    xfs.write_bytes
    xfs.read
    xfs.read_bytes
}

log mandatory on every 10 seconds {
    xfs.allocs
    xfs.block_map
    xfs.transactions
    xfs.log

}

[access]
disallow * : all;
allow localhost : enquire;

Replaying the PCP Log Archives

After recording metric data, you can replay the PCP log archives on the system in the following ways:
  • You can export the logs to text files and import them into spreadsheets by using PCP utilities such as pmdumptext, pmrep, or pmlogsummary.
  • You can replay the data in the PCP Charts application and use graphs to visualize the retrospective data alongside live data of the system. See Section 5.2.5, “Visual Tracing with PCP Charts”.
You can use the pmdumptext tool to view the log files. With pmdumptext, you can parse the selected PCP log archive and export the values into an ASCII table. The pmdumptext tool enables you to dump the entire archive log, or only select metric values from the log by specifying individual metrics on the command line.

Example 5.5. Displaying a Specific XFS Metric Log Information

For example, to show data on the xfs.perdev.log metric collected in an archive at a 5 second interval and display all headers: 
$ pmdumptext -t 5seconds -H -a 20170605 xfs.perdev.log.writes

Time local::xfs.perdev.log.writes["/dev/mapper/fedora-home"] local::xfs.perdev.log.writes["/dev/mapper/fedora-root"]
? 0.000 0.000
none count / second count / second
Mon Jun 5 12:28:45 ? ?
Mon Jun 5 12:28:50 0.000 0.000
Mon Jun 5 12:28:55 0.200 0.200
Mon Jun 5 12:29:00 6.800 1.000
For more information, see the pmdumptext(1) manual page, which is available from the pcp-doc package.

5.2.5. Visual Tracing with PCP Charts

To be able to use the graphical PCP Charts application, install the pcp-gui package: 
# yum install pcp-gui
You can use the PCP Charts application to plot performance metric values into graphs. The PCP Charts application allows multiple charts to be displayed simultaneously. The metrics are sourced from one or more live hosts with alternative options to use metric data from PCP log archives as a source of historical data. To launch PCP Charts from the command line, use the pmchart command.
After starting PCP Charts, the GUI appears:
The PCP Charts application
The PCP Charts application
The pmtime server settings are located at the bottom. The start and pause button allows you to control:
  • The interval in which PCP polls the metric data
  • The date and time for the metrics of historical data
Go to File → New Chart to select metric from both the local machine and remote machines by specifying their host name or address. Then, select performance metrics from the remote hosts. Advanced configuration options include the ability to manually set the axis values for the chart, and to manually choose the color of the plots.
There are multiple options to take images or record the views created in PCP Charts:
  • Click File → Export to save an image of the current view.
  • Click Record → Start to start a recording. Click Record → Stop to stop the recording. After stopping the recording, the recorded metrics are archived to be viewed later.
You can customize the PCP Charts interface to display the data from performance metrics in multiple ways, including:
  • line plot
  • bar graphs
  • utilization graphs
In PCP Charts, the main configuration file, known as the view, allows the metadata associated with one or more charts to be saved. This metadata describes all chart aspects, including the metrics used and the chart columns. You can create a custom view configuration, save it by clicking File → Save View, and load the view configuration later. For more information about view configuration files and their syntax, see the pmchart(1) manual page.

Example 5.6. Stacking Chart Graph in PCP Charts View Configuration

The example PCP Charts view configuration file describes a stacking chart graph showing the total number of bytes read and written to the given XFS file system loop1. 
#kmchart
version 1

chart title "Filesystem Throughput /loop1" style stacking antialiasing off
    plot legend "Read rate"   metric xfs.read_bytes   instance  "loop1"
    plot legend "Write rate"  metric xfs.write_bytes  instance  "loop1"