How to track slab allocations using perf

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 6 and above

Issue

  • How can I find out what is allocating in a slab cache

Resolution

  • The perf utility can be used to track allocations and capture process stacks making the allocations. For details on setting up perf, please review the knowledge-base article How do I collect performance data with perf which is readable on another machine?

  • The following script will perform the collection:

    #!/bin/sh
    [ $# -eq 0 ] && {
        echo "Usage: $0 <slab_cache_name> <timer>
    
    Note, the cache names can be found in '\proc\slabinfo'"
        exit 1
    }
    
    SLAB="$1"
    TIMER=30
    
    [ $# -eq 2 ] && { TIMER=$2; }
    
    grep -q ^"$SLAB" /proc/slabinfo || {
        echo "error: no '$SLAB' slab cache exists"; exit 2
        exit 2
    }
    
    UNAME_R=$(uname -r)
    echo "$UNAME_R" | grep -q el5 && { RHEL_VER=el5; }
    echo "$UNAME_R" | grep -q el6 && { RHEL_VER=el6; }
    echo "$UNAME_R" | grep -q el7 && { RHEL_VER=el7; }
    echo "$UNAME_R" | grep -q el8 && { RHEL_VER=el8; }
    
    perf probe -d kmem_cache_alloc* 2>/dev/null
    
    case $RHEL_VER in
        el5)
            perf probe kmem_cache_alloc 'cachep->name:string' 2>/dev/null    # RHEL 5
            ;;
        el6)
            perf probe kmem_cache_alloc 'cachep->name:string' 2>/dev/null    # RHEL 6
            ;;
        el7)
            perf probe kmem_cache_alloc      's->name:string' 2>/dev/null    # RHEL 7
            ;;
        el8)
            perf probe kmem_cache_alloc      's->name:string' 2>/dev/null    # RHEL 8
            ;;
        *)
            ;;
    esac
    
    grep -q 'probe/kmem_cache_alloc' /sys/kernel/debug/tracing/kprobe_events || {
        echo "error: failed to add the probe"
        exit 3
    }
    
    echo "collecting the data for $TIMER seconds, stand by..."
    perf record -a -g -e probe:kmem_cache_alloc --filter 'name == "'"$SLAB"\" sleep "$TIMER"
    perf probe -d kmem_cache_alloc* 2>/dev/null
    
    echo "creating the archive with debugging symbols..."
    rpm -q kernel-debuginfo-"$UNAME_R" >/dev/null \
    || echo "warning: package kernel-debuginfo-$UNAME_R is not installed"
    perf archive >/dev/null
    
    echo "done: please share both perf.data and" perf.data.tar.* "with Red Hat support for analysis"
    echo "note: if there is no perf.data.tar.* generated, $SLAB might not be in use during $TIMER seconds
          or $SLAB might not be used by kmem_cache_alloc function"
    

Sample Output

  • The output of the script will look as follows (note the 'kmalloc-64' cache is used as a filter. Also note that if no samples are collected for the cache that is used as a filter the perf archive command will not create a perf.data.tar.bz2 (or perf.data.tar.gz) file):

    # ./perf-record-all-slab.sh kmalloc-64 50
    collecting the data for 50 seconds, stand by...
    [ perf record: Woken up 35 times to write data ]
    [ perf record: Captured and wrote 10.050 MB perf.data (5 samples) ]
    creating the archive with debugging symbols...
    done: please share both perf.data and perf.data.tar.bz2 with Red Hat support for analysis
    note: if there is no perf.data.tar.* generated, kmalloc-64 might not be in use during 50 seconds
          or kmalloc-64 might not be used by kmem_cache_alloc function
    
  • The perf output can be reviewed by running perf script, sample output is below:

    perf 27021 [007] 2858133.918990: probe:kmem_cache_alloc: (ffffffff9d81c190) name="kmalloc-64"
                      41c191 kmem_cache_alloc (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      3e9b2a handle_pte_fault (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      3ec0fd handle_mm_fault (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      3e1a42 __get_user_pages (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      3e2464 get_user_pages_remote (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      448700 copy_strings.isra.18 (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      448804 copy_strings_kernel (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      44a188 do_execve_common.isra.24 (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      44a729 sys_execve (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      976368 stub_execve (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                       c5207 __execve (/usr/lib64/libc-2.17.so)
                       ad713 perf_evlist__prepare_workload (/usr/bin/perf)
                       3c551 cmd_record (/usr/bin/perf)
                       9a9cf [unknown] (/usr/bin/perf)
                       283b5 main (/usr/bin/perf)
                       223d5 __libc_start_main (/usr/lib64/libc-2.17.so)
    
    sleep 27021 [007] 2858133.919188: probe:kmem_cache_alloc: (ffffffff9d81c190) name="kmalloc-64"
                      41c191 kmem_cache_alloc (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      3e54b1 do_cow_fault (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      3e9624 handle_pte_fault (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      3ec0fd handle_mm_fault (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      9705e3 __do_page_fault (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      970915 do_page_fault (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      96c758 page_fault (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      587280 clear_user (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      4a46d9 padzero (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      4a6846 load_elf_binary (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      448b1a search_binary_handler (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      44a226 do_execve_common.isra.24 (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      44a729 sys_execve (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                      976368 stub_execve (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux)
                       c5207 __execve (/usr/lib64/libc-2.17.so)
    [...]
    

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.