RHEL5: system probe registration error when running systemtap script designed to track NFS stale file handle messages
Environment
- Red Hat Enterprise Linux 5.8
- systemtap-1.6-7.el5_8.x86_64
- kernel-2.6.18-308.16.1.el5.x86_64
- The following 3rd party kernel modules installed
stap_ptrace
vxodm
vxgms
amf
vxglm
vxfen
gab
llt
lin_tape
dmpaa
vxspec
vxio
vxdmp
vxportal
fdd
vxfs
Issue
When running the systemtap script provided in knowledge base article https://access.redhat.com/solutions/2674 warning messages are issued and no information is captured by it when the problem occurs.
Pass 1: parsed user script and 76 library script(s) using 145772virt/22524res/3004shr kb, in 120usr/0sys/128real ms.
Pass 2: analyzed script: 2499 probe(s), 13 function(s), 6 embed(s), 0 global(s) using 254400virt/69360res/4696shr kb, in 800usr/230sys/1036real ms.
Pass 3: using cached /root/.systemtap/cache/5b/stap_5b990eb3eaaad02722bdfc58eadb2c17_914729.c
Pass 4: using cached /root/.systemtap/cache/5b/stap_5b990eb3eaaad02722bdfc58eadb2c17_914729.ko
Pass 5: starting run.
Evidence collection started at Thu Jun 25 14:52:47 2015...
WARNING: probe kernel.function("fput@fs/file_table.c:143").return (address 0xffffffff8002cc71) registration error (rc -22)
WARNING: probe kernel.function("unlock_buffer@fs/buffer.c:79").return (address 0xffffffff8003b431) registration error (rc -22)
WARNING: probe kernel.function("__lock_buffer@fs/buffer.c:72").return (address 0xffffffff8003fa93) registration error (rc -22)
WARNING: probe kernel.function("dget_locked@fs/dcache.c:281").return (address 0xffffffff80041fe8) registration error (rc -22)
WARNING: probe kernel.function("writeback_acquire@fs/fs-writeback.c:690").return (address 0xffffffff80049bf7) registration error (rc -22)
WARNING: probe kernel.function("get_inotify_watch@fs/inotify.c:103").return (address 0xffffffff800fb760) registration error (rc -22)
Resolution
- Either remove the third party modules or modify the systemtap script to be more restrictive, and not probe all files with wildcards.
Root Cause
- The presence of the third party modules are believed to cause this issue.
- Internal testing revealed no errors with the systemtap script when testing with identical kernel and systemtap modules.
Diagnostic Steps
debuginfo packages have been installed manually instead of using stap-prep.
Running that command afterwards returned nothing, meaning that nothing was missing and the environment was ready.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments