How to monitor slab allocations using eBPF/BCC scripts

Resolution

Note At present, eBPF can not be used to capture the name of a slab cache being allocated from, the PID of the proces doing the allocation, and the kernel stack of the process all within the same sample. Allocation behavior trends within processes can be inferred by capturing both of the following simultaneously.

Counting allocation by slab cache

• The eBPF/BCC slabratetop script shows the rate of allocations and total bytes from the kernel memory allocation caches (with slab or slub), in a top-like display that refreshes.
• It uses kernel dynamic tracing within the kmem_cache_alloc() function.

• To use eBPF slabratetop script , first install bcc, bcc-tools, and kernel-devel packages.

  # yum -y install bcc bcc-tools kernel-devel-`uname -r`
  

• Example of the slabratetop script usage:

  # /usr/share/bcc/tools/slabratetop
  17:31:20 loadavg: 0.02 0.01 0.00 1/165 31776
  
  CACHE                            ALLOCS      BYTES
  names_cache                          17      69632
  vm_area_struct                      220      51040
  pid                                 212      13568
  anon_vma                            107       9416
  proc_inode_cache                     10       7040
  shmem_inode_cache                     8       5888
  filp                                 21       5376
  dentry                               19       3648
  mm_struct                             2       2304
  sighand_cache                         1       2112
  cred_jar                              8       1536
  signal_cache                          1       1088
  skbuff_head_cache                     4       1024
  Acpi-Namespace                       20        800
  files_cache                           1        704
  inode_cache                           1        632
  selinux_file_security                21        336
  ebitmap_node                          3        192
  sigqueue                              1        160
  seq_file                              1        128
  Detaching...
  

• Please see man 8 slabratetop for full options.

Tracing processes performing allocations

• The trace eBPF script can also be used to trace calls to kmem_cache_alloc. The below example collects the backtraces of all processes calling kmem_cache_alloc for 5 seconds;

  # /usr/share/bcc/tools/trace -K 't:kmem:kmem_cache_alloc' > trace 2> /dev/null & trace_pid="$!" && sleep 5 && kill -sigkill "$trace_pid"
  [1] 9193
  # 
  [1]+  Killed                  /usr/share/bcc/tools/trace -K 't:kmem:kmem_cache_alloc' > trace 2> /dev/null
  

• From here, the trace file can be inspected:

  # less trace
  PID     TID     COMM            FUNC             
  9120    9120    kworker/0:2     kmem_cache_alloc 
          b'kmem_cache_alloc+0x1a8 [kernel]'
          b'kmem_cache_alloc+0x1a8 [kernel]'
          b'__d_alloc+0x22 [kernel]'
          b'd_alloc_pseudo+0xa [kernel]'
          b'__shmem_file_setup.part.43+0x75 [kernel]'
          b'drm_gem_object_init+0x26 [drm]'
          b'qxl_bo_create+0x6f [qxl]'
          b'qxl_alloc_bo_reserved+0x3c [qxl]'
          b'qxl_draw_dirty_fb+0x114 [qxl]'
          b'qxl_framebuffer_surface_dirty+0xac [qxl]'
          b'drm_fb_helper_dirty_work+0x168 [drm_kms_helper]'
          b'process_one_work+0x1a7 [kernel]'
          b'worker_thread+0x30 [kernel]'
          b'kthread+0x116 [kernel]'
          b'ret_from_fork+0x1f [kernel]'
  
  9120    9120    kworker/0:2     kmem_cache_alloc 
          b'kmem_cache_alloc+0x1a8 [kernel]'
          b'kmem_cache_alloc+0x1a8 [kernel]'
          b'shmem_alloc_inode+0x16 [kernel]'
          b'alloc_inode+0x1b [kernel]'
          b'new_inode_pseudo+0xc [kernel]'
          b'new_inode+0x12 [kernel]'
          b'shmem_get_inode+0x49 [kernel]'
          b'__shmem_file_setup.part.43+0xa6 [kernel]'
          b'drm_gem_object_init+0x26 [drm]'
          b'qxl_bo_create+0x6f [qxl]'
  [...]