Have you tried SystemTap?

SystemTap is designed to eliminate the need for the developer to go through the tedious instrument, recompile, install, and restart sequence required to collect data on the operation of the code. SystemTap provides a simple command line interface and scripting language for writing instrumentation.

Our lead SystemTap Developer has an excellent 5 minute video that walk you through it here:  https://access.redhat.com/knowledge/videos/systemtap-overview

In addition, here's a really good and detailed white paper written on the subject as well:  https://access.redhat.com/knowledge/techbriefs/instrumenting-linux-kernel-systemtap.

Please share your experience and/or questions with SystemTap. 

If you have great usage examples or learning tools on SystemTap, please do share them.