PCP does not record mem.slabinfo* data or some other portions of memory data

Solution Verified - Updated -

Environment

  • Reported on, but not limited to, Red Hat Enterprise Linux 8.3

  • PCP packages versioned prior to, but not including, pcp-5.3.5-8.el8.

Issue

  • When reviewing memory statistics recorded with PCP, slab performance metrics and other memory related data was not collected.

Resolution

  • Please update the PCP packages to version pcp-5.3.5-8.el8 or later, as RHBA-2022:1765 resolved this issue.

  • If it's not possible to update the PCP packages to pcp-5.3.5-8.el8 or later, a potential unsupported workaround is to perform the following steps. Note this this has been reported to work, but is not guaranteed.

    1. Edit /etc/pcp/pmlogger/control.d/local and make the last line look like -U root:

      LOCALHOSTNAME   y   n   PCP_ARCHIVE_DIR/LOCALHOSTNAME   -r -T24h10m -c config.default -v 100Mb -U root

    2. Restart pmlogger with the following command:

      # systemctl restart pmlogger

    3. Verify that the pmlogger runs as the root user and not as pcp.

      ~~~
# ls -l
total 79696
-rw-r--r--. 1 pcp  pcp   7218848 May 20 21:51 20210520.19.12.0.xz
-rw-r--r--. 1 pcp  pcp   6925308 May 21 00:33 20210520.19.12.1.xz
-rw-r--r--. 1 pcp  pcp   6900852 May 21 03:16 20210520.19.12.2.xz
-rw-r--r--. 1 pcp  pcp   6911976 May 21 05:58 20210520.19.12.3.xz
-rw-r--r--. 1 root root 30283152 May 21 06:44 20210520.19.12.4
-rw-r--r--. 1 root root    58372 May 21 06:44 20210520.19.12.index
-rw-r--r--. 1 root root 10092349 May 21 06:44 20210520.19.12.meta
-rw-r--r--. 1 root root      254 May 20 19:12 Latest
-rw-r--r--. 1 root root    32732 May 21 05:58 pmlogger.log
~~~

Root Cause

  • In order to access the mem.slabinfo metric values the PMAPI client process must be authenticated (with pmcd/pmdalinux) as 'root'.

  • This is because of the potential (i.e. previous kernel CVEs) for information exposure from this metric data, so the kernel exports it as:

    $ cat /proc/slabinfo 
cat: /proc/slabinfo: Permission denied

$ ls -ld /proc/slabinfo
-r--------. 1 root root 0 May 21 08:23 /proc/slabinfo

  • In versions of PCP prior to pcp-5.3.5-8.el8 the permissions were respected, and root access was required to read these metrics.

  • This difference is observable when running pminfo -f mem.slabinfo as opposed to sudo pminfo -f mem.slabinfo.

Diagnostic Steps

  • Reviewing data previously recorded with PCP versioned 5.1.1-4.el8_3, we see that some of the memory statistics were not recorded.

    $ pmrep :sar-r -z -a 20210512.0.xz | head
        kbmemfree kbmemavai kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
00:10:02          N/A       N/A       N/A       N/A       N/A       N/A       N/A       N/A       N/A       N/A       N/A
00:10:03          N/A       N/A       N/A       N/A        36   5712308       N/A       N/A       N/A       N/A       N/A
00:10:04          N/A       N/A       N/A       N/A        36   5712308       N/A       N/A       N/A       N/A       N/A
00:10:05          N/A       N/A       N/A       N/A        36   5712308       N/A       N/A       N/A       N/A       N/A

  • The slabinfo did not collect at all:

    $ pmrep :slabinfo -z -a 20210512.0.xz
Invalid metric mem.slabinfo.objects.active (PM_ERR_INDOM_LOG Instance domain identifier not defined in the PCP archive log).3

  • However, based on the pmlogger and pminfo the slabinfo should be here:

    $ pminfo -a 20210512.0.xz  | grep mem.slabinfo*
mem.slabinfo.objects.size
mem.slabinfo.objects.active
mem.slabinfo.objects.total
mem.slabinfo.slabs.active
mem.slabinfo.slabs.total
mem.slabinfo.slabs.pages_per_slab
mem.slabinfo.slabs.objects_per_slab
mem.slabinfo.slabs.total_size
  • Component
  • pcp

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments