Red Hat Training

A Red Hat training course is available for RHEL 8

Chapter 9. Performance analysis of XFS with PCP

The XFS PMDA ships as part of the pcp package and is enabled by default during the installation. It is used to gather performance metric data of XFS file systems in Performance Co-Pilot (PCP).

This section describes how to analyze XFS file system’s performance using PCP.

9.1. Installing XFS PMDA manually

If the XFS PMDA is not listed in the pcp configuration output, install the PMDA agent manually.

This procedure describes how to manually install the PMDA agent.

Prerequisites

Procedure

  1. Navigate to the xfs directory:

    # cd /var/lib/pcp/pmdas/xfs/
  2. Install the XFS PMDA manually:

    xfs]# ./Install
    
    You will need to choose an appropriate configuration for install of
    the “xfs” Performance Metrics Domain Agent (PMDA).
    
      collector     collect performance statistics on this system
      monitor       allow this system to monitor local and/or remote systems
      both          collector and monitor configuration for this system
    
    Please enter c(ollector) or m(onitor) or (both) [b]
    Updating the Performance Metrics Name Space (PMNS) ...
    Terminate PMDA if already installed ...
    Updating the PMCD control file, and notifying PMCD ...
    Waiting for pmcd to terminate ...
    Starting pmcd ...
    Check xfs metrics have appeared ... 149 metrics and 149 values
  3. Select the intended PMDA role by entering c for collector, m for monitor, or b for both. The PMDA installation script prompts you to specify one of the following PMDA roles:

    • The collector role allows the collection of performance metrics on the current system
    • The monitor role allows the system to monitor local systems, remote systems, or both

      The default option is both collector and monitor, which allows the XFS PMDA to operate correctly in most scenarios.

Verification steps

  • Verify that the pmcd process is running on the host and the XFS PMDA is listed as enabled in the configuration:

    # pcp
    
    Performance Co-Pilot configuration on workstation:
    
    platform: Linux workstation 4.18.0-80.el8.x86_64 #1 SMP Wed Mar 13 12:02:46 UTC 2019 x86_64
    hardware: 12 cpus, 2 disks, 1 node, 36023MB RAM
    timezone: CEST-2
    services: pmcd
    pmcd: Version 4.3.0-1, 8 agents
    pmda: root pmcd proc xfs linux mmv kvm jbd2

Additional resources

9.2. Examining XFS performance metrics with pminfo

PCP enables XFS PMDA to allow the reporting of certain XFS metrics per each of the mounted XFS file systems. This makes it easier to pinpoint specific mounted file system issues and evaluate performance.

The pminfo command provides per-device XFS metrics for each mounted XFS file system.

This procedure displays a list of all available metrics provided by the XFS PMDA.

Prerequisites

Procedure

  • Display the list of all available metrics provided by the XFS PMDA:

    # pminfo xfs
  • Display information for the individual metrics. The following examples examine specific XFS read and write metrics using the pminfo tool:

    • Display a short description of the xfs.write_bytes metric:

      # pminfo --oneline xfs.write_bytes
      
      xfs.write_bytes [number of bytes written in XFS file system write operations]
    • Display a long description of the xfs.read_bytes metric:

      # pminfo --helptext xfs.read_bytes
      
      xfs.read_bytes
      Help:
      This is the number of bytes read via read(2) system calls to files in
      XFS file systems. It can be used in conjunction with the read_calls
      count to calculate the average size of the read operations to file in
      XFS file systems.
    • Obtain the current performance value of the xfs.read_bytes metric:

      # pminfo --fetch xfs.read_bytes
      
      xfs.read_bytes
          value 4891346238
    • Obtain per-device XFS metrics with pminfo:

      # pminfo --fetch --oneline xfs.perdev.read xfs.perdev.write
      
      xfs.perdev.read [number of XFS file system read operations]
      inst [0 or "loop1"] value 0
      inst [0 or "loop2"] value 0
      
      xfs.perdev.write [number of XFS file system write operations]
      inst [0 or "loop1"] value 86
      inst [0 or "loop2"] value 0

Additional resources

9.3. Resetting XFS performance metrics with pmstore

With PCP, you can modify the values of certain metrics, especially if the metric acts as a control variable, such as the xfs.control.reset metric. To modify a metric value, use the pmstore tool.

This procedure describes how to reset XFS metrics using the pmstore tool.

Prerequisites

Procedure

  1. Display the value of a metric:

    $ pminfo -f xfs.write
    
    xfs.write
        value 325262
  2. Reset all the XFS metrics:

    # pmstore xfs.control.reset 1
    
    xfs.control.reset old value=0 new value=1

Verification steps

  • View the information after resetting the metric:

    $ pminfo --fetch xfs.write
    
    xfs.write
        value 0

Additional resources

9.4. PCP metric groups for XFS

The following table describes the available PCP metric groups for XFS.

Table 9.1. Metric groups for XFS

Metric Group

Metrics provided

xfs.*

General XFS metrics including the read and write operation counts, read and write byte counts. Along with counters for the number of times inodes are flushed, clustered and number of failure to cluster.

xfs.allocs.*

xfs.alloc_btree.*

Range of metrics regarding the allocation of objects in the file system, these include number of extent and block creations/frees. Allocation tree lookup and compares along with extend record creation and deletion from the btree.

xfs.block_map.*

xfs.bmap_tree.*

Metrics include the number of block map read/write and block deletions, extent list operations for insertion, deletions and lookups. Also operations counters for compares, lookups, insertions and deletion operations from the blockmap.

xfs.dir_ops.*

Counters for directory operations on XFS file systems for creation, entry deletions, count of “getdent” operations.

xfs.transactions.*

Counters for the number of meta-data transactions, these include the count for the number of synchronous and asynchronous transactions along with the number of empty transactions.

xfs.inode_ops.*

Counters for the number of times that the operating system looked for an XFS inode in the inode cache with different outcomes. These count cache hits, cache misses, and so on.

xfs.log.*

xfs.log_tail.*

Counters for the number of log buffer writes over XFS file sytems includes the number of blocks written to disk. Metrics also for the number of log flushes and pinning.

xfs.xstrat.*

Counts for the number of bytes of file data flushed out by the XFS flush deamon along with counters for number of buffers flushed to contiguous and non-contiguous space on disk.

xfs.attr.*

Counts for the number of attribute get, set, remove and list operations over all XFS file systems.

xfs.quota.*

Metrics for quota operation over XFS file systems, these include counters for number of quota reclaims, quota cache misses, cache hits and quota data reclaims.

xfs.buffer.*

Range of metrics regarding XFS buffer objects. Counters include the number of requested buffer calls, successful buffer locks, waited buffer locks, miss_locks, miss_retries and buffer hits when looking up pages.

xfs.btree.*

Metrics regarding the operations of the XFS btree.

xfs.control.reset

Configuration metrics which are used to reset the metric counters for the XFS stats. Control metrics are toggled by means of the pmstore tool.

9.5. Per-device PCP metric groups for XFS

The following table describes the available per-device PCP metric group for XFS.

Table 9.2. Per-device PCP metric groups for XFS

Metric Group

Metrics provided

xfs.perdev.*

General XFS metrics including the read and write operation counts, read and write byte counts. Along with counters for the number of times inodes are flushed, clustered and number of failure to cluster.

xfs.perdev.allocs.*

xfs.perdev.alloc_btree.*

Range of metrics regarding the allocation of objects in the file system, these include number of extent and block creations/frees. Allocation tree lookup and compares along with extend record creation and deletion from the btree.

xfs.perdev.block_map.*

xfs.perdev.bmap_tree.*

Metrics include the number of block map read/write and block deletions, extent list operations for insertion, deletions and lookups. Also operations counters for compares, lookups, insertions and deletion operations from the blockmap.

xfs.perdev.dir_ops.*

Counters for directory operations of XFS file systems for creation, entry deletions, count of “getdent” operations.

xfs.perdev.transactions.*

Counters for the number of meta-data transactions, these include the count for the number of synchronous and asynchronous transactions along with the number of empty transactions.

xfs.perdev.inode_ops.*

Counters for the number of times that the operating system looked for an XFS inode in the inode cache with different outcomes. These count cache hits, cache misses, and so on.

xfs.perdev.log.*

xfs.perdev.log_tail.*

Counters for the number of log buffer writes over XFS filesytems includes the number of blocks written to disk. Metrics also for the number of log flushes and pinning.

xfs.perdev.xstrat.*

Counts for the number of bytes of file data flushed out by the XFS flush deamon along with counters for number of buffers flushed to contiguous and non-contiguous space on disk.

xfs.perdev.attr.*

Counts for the number of attribute get, set, remove and list operations over all XFS file systems.

xfs.perdev.quota.*

Metrics for quota operation over XFS file systems, these include counters for number of quota reclaims, quota cache misses, cache hits and quota data reclaims.

xfs.perdev.buffer.*

Range of metrics regarding XFS buffer objects. Counters include the number of requested buffer calls, successful buffer locks, waited buffer locks, miss_locks, miss_retries and buffer hits when looking up pages.

xfs.perdev.btree.*

Metrics regarding the operations of the XFS btree.