How to track slab allocations using perf
Environment
- Red Hat Enterprise Linux 6 and above
Issue
- How can I find out what is allocating in a slab cache
Resolution
-
The
perfutility can be used to track allocations and capture process stacks making the allocations. For details on setting upperf, please review the knowledge-base article How do I collect performance data with perf which is readable on another machine? -
The following script will perform the collection:
#!/bin/bash
[ $# -eq 0 ] && {
echo "Usage: $0 <slab_cache_name> [timer]\n\nNote, the cache names can be found in '/proc/slabinfo'."
exit 1
}
SLAB=$1
TIMER=$2
[ "$TIMER" -ge 0 ] 2>/dev/null || TIMER=10
grep -q "^$SLAB" /proc/slabinfo || {
echo "Error: No $SLAB slab cache exists!"
exit 2
}
DEBUGINFO=kernel-debuginfo-$(uname -r)
rpm -q $DEBUGINFO >/dev/null || echo "Warning: Package $DEBUGINFO is not installed!"
echo "Preparing kernel probes..."
case $(uname -r | grep -o el[5-9]) in
el5 | el6)
perf probe -d kmem_cache_alloc* 2>/dev/null
perf probe kmem_cache_alloc 'cachep->name:string' 2>/dev/null
;;
el7 | el8 | el9)
perf probe -d kmem_cache_alloc* 2>/dev/null
perf probe kmem_cache_alloc 's->name:string' 2>/dev/null
;;
*)
echo "Error: Unknown kernel version!"
exit 2
esac
grep -q probe/kmem_cache_alloc /sys/kernel/debug/tracing/kprobe_events || {
echo "Error: Failed to add the probe!"
exit 3
}
echo "Collecting the data for $TIMER seconds, stand by..."
perf record -a -g -e probe:kmem_cache_alloc --filter "name == \"$SLAB\"" sleep $TIMER
perf probe -d kmem_cache_alloc* 2>/dev/null
echo "Creating the archive with debugging symbols..."
perf archive >/dev/null
echo "Done: Please share both perf.data and" perf.data.tar.* "with Red Hat support for analysis."
echo "Note: If there is no perf.data.tar.* generated, \
the $SLAB slab might not have been used during the $TIMER seconds or \
the $SLAB cache might not be used by kmem_cache_alloc function at all."
- Note:- If you face
Error: Failed to add the probe!error while executing the above script please make sure kernel-debuginfo and kernel-debuginfo-common packages are installed and compatible with the installed kernel. Even if perf is not installed on the server you will encounter the same error (error: failed to add the probe).
Sample Output
-
The output of the script will look as follows (note the 'kmalloc-64' cache is used as a filter. Also note that if no samples are collected for the cache that is used as a filter the
perf archivecommand will not create a perf.data.tar.bz2 (or perf.data.tar.gz) file):# ./perf-record-all-slab.sh kmalloc-64 50 collecting the data for 50 seconds, stand by... [ perf record: Woken up 35 times to write data ] [ perf record: Captured and wrote 10.050 MB perf.data (5 samples) ] creating the archive with debugging symbols... done: please share both perf.data and perf.data.tar.bz2 with Red Hat support for analysis note: if there is no perf.data.tar.* generated, kmalloc-64 might not be in use during 50 seconds or kmalloc-64 might not be used by kmem_cache_alloc function -
The perf output can be reviewed by running
perf script, sample output is below:perf 27021 [007] 2858133.918990: probe:kmem_cache_alloc: (ffffffff9d81c190) name="kmalloc-64" 41c191 kmem_cache_alloc (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 3e9b2a handle_pte_fault (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 3ec0fd handle_mm_fault (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 3e1a42 __get_user_pages (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 3e2464 get_user_pages_remote (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 448700 copy_strings.isra.18 (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 448804 copy_strings_kernel (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 44a188 do_execve_common.isra.24 (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 44a729 sys_execve (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 976368 stub_execve (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) c5207 __execve (/usr/lib64/libc-2.17.so) ad713 perf_evlist__prepare_workload (/usr/bin/perf) 3c551 cmd_record (/usr/bin/perf) 9a9cf [unknown] (/usr/bin/perf) 283b5 main (/usr/bin/perf) 223d5 __libc_start_main (/usr/lib64/libc-2.17.so) sleep 27021 [007] 2858133.919188: probe:kmem_cache_alloc: (ffffffff9d81c190) name="kmalloc-64" 41c191 kmem_cache_alloc (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 3e54b1 do_cow_fault (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 3e9624 handle_pte_fault (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 3ec0fd handle_mm_fault (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 9705e3 __do_page_fault (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 970915 do_page_fault (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 96c758 page_fault (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 587280 clear_user (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 4a46d9 padzero (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 4a6846 load_elf_binary (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 448b1a search_binary_handler (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 44a226 do_execve_common.isra.24 (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 44a729 sys_execve (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) 976368 stub_execve (/usr/lib/debug/lib/modules/3.10.0-957.10.1.el7.x86_64/vmlinux) c5207 __execve (/usr/lib64/libc-2.17.so) [...]
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments