RHEL5: system probe registration error when running systemtap script designed to track NFS stale file handle messages

Solution In Progress - Updated -

Environment

  • Red Hat Enterprise Linux 5.8
  • systemtap-1.6-7.el5_8.x86_64
  • kernel-2.6.18-308.16.1.el5.x86_64
  • The following 3rd party kernel modules installed
  stap_ptrace
  vxodm
  vxgms
  amf
  vxglm
  vxfen
  gab
  llt
  lin_tape
  dmpaa
  vxspec
  vxio
  vxdmp
  vxportal
  fdd
  vxfs

Issue

When running the systemtap script provided in knowledge base article https://access.redhat.com/solutions/2674 warning messages are issued and no information is captured by it when the problem occurs.

Pass 1: parsed user script and 76 library script(s) using 145772virt/22524res/3004shr kb, in 120usr/0sys/128real ms.
Pass 2: analyzed script: 2499 probe(s), 13 function(s), 6 embed(s), 0 global(s) using 254400virt/69360res/4696shr kb, in 800usr/230sys/1036real ms.
Pass 3: using cached /root/.systemtap/cache/5b/stap_5b990eb3eaaad02722bdfc58eadb2c17_914729.c
Pass 4: using cached /root/.systemtap/cache/5b/stap_5b990eb3eaaad02722bdfc58eadb2c17_914729.ko
Pass 5: starting run.
Evidence collection started at Thu Jun 25 14:52:47 2015...
WARNING: probe kernel.function("fput@fs/file_table.c:143").return (address 0xffffffff8002cc71) registration error (rc -22)
WARNING: probe kernel.function("unlock_buffer@fs/buffer.c:79").return (address 0xffffffff8003b431) registration error (rc -22)
WARNING: probe kernel.function("__lock_buffer@fs/buffer.c:72").return (address 0xffffffff8003fa93) registration error (rc -22)
WARNING: probe kernel.function("dget_locked@fs/dcache.c:281").return (address 0xffffffff80041fe8) registration error (rc -22)
WARNING: probe kernel.function("writeback_acquire@fs/fs-writeback.c:690").return (address 0xffffffff80049bf7) registration error (rc -22)
WARNING: probe kernel.function("get_inotify_watch@fs/inotify.c:103").return (address 0xffffffff800fb760) registration error (rc -22)

Resolution

  • Either remove the third party modules or modify the systemtap script to be more restrictive, and not probe all files with wildcards.

Root Cause

  • The presence of the third party modules are believed to cause this issue.
  • Internal testing revealed no errors with the systemtap script when testing with identical kernel and systemtap modules.

Diagnostic Steps

debuginfo packages have been installed manually instead of using stap-prep.
Running that command afterwards returned nothing, meaning that nothing was missing and the environment was ready.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments