How to use systemtap for probing threaded user space applications?

Issue

• If a multi-threaded program with the main thread not going through at least one quiesce after systemtap/uprobes tracing starts, then systemtap is not able to detect the events.

• But it works when the application and stap script is ran concurrently. It is demonstrated below.

• Compile the program pthread.cpp as shown below.

g++ -m32 -g -o pthread  pthread.cpp -lpthread

#include <pthread.h>
#include <stdio.h>
#include <unistd.h>

int square (int x)
{
  return (x * x);
}

void *my_thread(void *arg)
{
    int x = 0;
    int sqr;

    while (x < 100) {

        sqr = square(x);

        sleep(1);

        x++;

    }

    return NULL;

}

int main()
{

    pthread_t thread_id;

    if (pthread_create(&thread_id, NULL, my_thread, NULL)) {

        fprintf(stderr, "Error creating thread\n");
        return 1;

    }

    if (pthread_join(thread_id, NULL)) {

         fprintf(stderr, "Error joining thread\n");
         return 2;

    }


    return 0;

} 

• Consider below systemtap script, pthread.stp

#! /usr/bin/env stap

probe process("./pthread").function("square") 
{ 
    printf("=> %s(%s)\n", probefunc(), $$parms);  
}

probe process("./pthread").function("square").return
{ 
    printf("<= %s\n", probefunc());  
}

probe process("./pthread").statement("*@pthread.cpp:19") 
{ 
    printf("   x=%d, sqr=%d\n\n", $x, $sqr);  
}

• For probing the application pthread, execute it as shown below.

stap -c ./pthread pthread.stp

• Sample output for the above command

[root@dhcp209-221 stap]# stap -c ./pthread pthread.stp 
=> square(x=0x0)
<= square
   x=0, sqr=0

=> square(x=0x1)
<= square
   x=1, sqr=1

=> square(x=0x2)
<= square
   x=2, sqr=4

=> square(x=0x3)
<= square
   x=3, sqr=9

=> square(x=0x4)
<= square
   x=4, sqr=16

=> square(x=0x5)
<= square
   x=5, sqr=25

=> square(x=0x6)
<= square
   x=6, sqr=36

=> square(x=0x7)
<= square
   x=7, sqr=49