RHEL5.8 - systemtap script fails to execute erroring with "file exists"

Solution Unverified - Updated -

Environment

  • RedHat Enterprise Linux 5.8

Issue

  • Customer was provided a systemtap script to investigate a problem and when trying to execute, received the following:
Error inserting module '/tmp/stappxRaAi/stap_716e52fe561e6a0fc75d400494876847_24172.ko': File exists
WARNING: /usr/bin/staprun exited with status: 1
Pass 5: run failed.  Try again with another '--vp 00001' option.

Resolution

  • Check if "stap_716e52fe561e6a0fc75d400494876847_24172" is already loaded, unload it and then execute the stap command
# lsmod | grep stap_716e52fe561e6a0fc75d400494876847_24172
# rmmod stap_716e52fe561e6a0fc75d400494876847_24172
  • Ensure that all files are removed from /tmp. Note /tmp in the messages reported:
Running /usr/bin/staprun -v -R /tmp/stapPahL2e/stap_716e52fe561e6a0fc75d400494876847_24172.ko
Error inserting module '/tmp/stapPahL2e/stap_716e52fe561e6a0fc75d400494876847_24172.ko': File exists

systemtap builds a .ko file in /tmp from the source stap script. The customer had numerous stap scripts and directories in /tmp which had accumulated over a long period of time.

  • Additionally, run stap scripts from a /home, /root or other directory but avoid copying the script and other files into /tmp and executing them from there.

Root Cause

  • Customer was moving the stap script into /tmp which was not a temporary directory any more and contained permanent files from numerous previous stap creations amongst other things.

Diagnostic Steps

  • Ensure the correct packages are installed:

Note. The first 3 items (kernel-*) should all be the same version as the running kernel on the system

1) kernel-devel
2) kernel-debuginfo
3) kernel-debuginfo-common
4) gcc
5) systemtap

  • To ensure that stap is actually installed correctly, run the following command:
# stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'

Pass 1: parsed user script and 86 library script(s) using 195572virt/24252res/3068shr/21492data kb, in 80usr/0sys/92real ms.
Pass 2: analyzed script: 1 probe(s), 1 function(s), 3 embed(s), 0 global(s) using 425568virt/123804res/8280shr/113420data kb, in 940usr/220sys/48134real ms.
Pass 3: translated to C into "/tmp/stapZB0pBq/stap_8eb3a5f6f5f6e6744e3abe53050e094e_1506_src.c" using 415740virt/119684res/6640shr/113420data kb, in 0usr/60sys/314real ms.
Pass 4: compiled C into "stap_8eb3a5f6f5f6e6744e3abe53050e094e_1506.ko" in 1120usr/310sys/56071real ms.
Pass 5: starting run.
read performed
Pass 5: run completed in 10usr/20sys/349real ms.
  • Alternatively/additionally, try creating a .ko of the affected stap script and pausing at phase 4 of the process (sigkill2.stp is just a sample script for the purpose of showing the steps involved):
# stap -p4 sigkill2.stp 
/root/.systemtap/cache/09/stap_09a95aadb4edea83d2985e768da7d201_23119.ko
  • And now just run the .ko
# staprun /root/.systemtap/cache/09/stap_09a95aadb4edea83d2985e768da7d201_23119.ko
Fri Sep 20 17:41:32 2013 SIGKILL sent to watch (pid:24327) - by bash (pid:24331) uid:500
  • If none of these steps are successful, add -vvv to the stap script command line to obtain a very verbose sequence of steps for each phase of the build and execution. Submit all the outputs from the suggestions in this Solution for our evaluation.
# stap -vvv systemtap.stp

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.