How to monitor slab allocations using eBPF/BCC scripts

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 8
  • bcc-tools

Issue

  • How to monitor the kernel SLAB/SLUB memory allocations using eBPF/BCC script?

Resolution

Note At present, eBPF can not be used to capture the name of a slab cache being allocated from, the PID of the proces doing the allocation, and the kernel stack of the process all within the same sample. Allocation behavior trends within processes can be inferred by capturing both of the following simultaneously.

Counting allocation by slab cache

  • The eBPF/BCC slabratetop script shows the rate of allocations and total bytes from the kernel memory allocation caches (with slab or slub), in a top-like display that refreshes.
  • It uses kernel dynamic tracing within the kmem_cache_alloc() function.
  • To use eBPF slabratetop script , first install bcc, bcc-tools, and kernel-devel packages.

    # yum -y install bcc bcc-tools kernel-devel-`uname -r`
    
  • Example of the slabratetop script usage:

    # /usr/share/bcc/tools/slabratetop
    17:31:20 loadavg: 0.02 0.01 0.00 1/165 31776
    
    CACHE                            ALLOCS      BYTES
    names_cache                          17      69632
    vm_area_struct                      220      51040
    pid                                 212      13568
    anon_vma                            107       9416
    proc_inode_cache                     10       7040
    shmem_inode_cache                     8       5888
    filp                                 21       5376
    dentry                               19       3648
    mm_struct                             2       2304
    sighand_cache                         1       2112
    cred_jar                              8       1536
    signal_cache                          1       1088
    skbuff_head_cache                     4       1024
    Acpi-Namespace                       20        800
    files_cache                           1        704
    inode_cache                           1        632
    selinux_file_security                21        336
    ebitmap_node                          3        192
    sigqueue                              1        160
    seq_file                              1        128
    Detaching...
    
  • Please see man 8 slabratetop for full options.

Tracing processes performing allocations

  • The trace eBPF script can also be used to trace calls to kmem_cache_alloc. The below example collects the backtraces of all processes calling kmem_cache_alloc for 5 seconds;

    # /usr/share/bcc/tools/trace -K 't:kmem:kmem_cache_alloc' > trace 2> /dev/null & trace_pid="$!" && sleep 5 && kill -sigkill "$trace_pid"
    [1] 9193
    # 
    [1]+  Killed                  /usr/share/bcc/tools/trace -K 't:kmem:kmem_cache_alloc' > trace 2> /dev/null
    
  • From here, the trace file can be inspected:

    # less trace
    PID     TID     COMM            FUNC             
    9120    9120    kworker/0:2     kmem_cache_alloc 
            b'kmem_cache_alloc+0x1a8 [kernel]'
            b'kmem_cache_alloc+0x1a8 [kernel]'
            b'__d_alloc+0x22 [kernel]'
            b'd_alloc_pseudo+0xa [kernel]'
            b'__shmem_file_setup.part.43+0x75 [kernel]'
            b'drm_gem_object_init+0x26 [drm]'
            b'qxl_bo_create+0x6f [qxl]'
            b'qxl_alloc_bo_reserved+0x3c [qxl]'
            b'qxl_draw_dirty_fb+0x114 [qxl]'
            b'qxl_framebuffer_surface_dirty+0xac [qxl]'
            b'drm_fb_helper_dirty_work+0x168 [drm_kms_helper]'
            b'process_one_work+0x1a7 [kernel]'
            b'worker_thread+0x30 [kernel]'
            b'kthread+0x116 [kernel]'
            b'ret_from_fork+0x1f [kernel]'
    
    9120    9120    kworker/0:2     kmem_cache_alloc 
            b'kmem_cache_alloc+0x1a8 [kernel]'
            b'kmem_cache_alloc+0x1a8 [kernel]'
            b'shmem_alloc_inode+0x16 [kernel]'
            b'alloc_inode+0x1b [kernel]'
            b'new_inode_pseudo+0xc [kernel]'
            b'new_inode+0x12 [kernel]'
            b'shmem_get_inode+0x49 [kernel]'
            b'__shmem_file_setup.part.43+0xa6 [kernel]'
            b'drm_gem_object_init+0x26 [drm]'
            b'qxl_bo_create+0x6f [qxl]'
    [...]
    

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.