As an engineer who works on performance tools at Red Hat, I often get seemingly simple questions along the lines of, "How do I get performance tool X to collect Y data?" Unfortunately, many times the answer is that "tool X does not measure Y." This leads to a dicussion about the performance problem being investigated. With additional background information, it becomes much easier to suggest more promising tools and techniques to get the desired measurements.
Given the number of performance tools and complexity of Linux, it is easy for developers and system administrators to end up trying to press what tools they know about onto tasks the tools are really not suited for. I have been guilty of this, too. Part of this tool-centric bias might be an effect of documentation being focused on individual tools rather than the tasks people want to accomplish.
Here at Red Hat, we are working to make it easier for people to measure and understand machine performance with a Performance Measurement Cookbook. Rather than being focused on the tools themselves, the cookbook tasks (or recipes) are focused on how to get the data that you need to understand what is happening on your system.
An example of the task-oriented nature of the Performance Measurement Cookbook is the Determine whether an application has poor cache performance recipe. It shows you precisely how to use the
perf utily to compute the cache miss rate for an application. If the cache miss rate is high, you can use the recipe to locate the places in the code where the cache misses occur. As you gain experience, you can modify this recipe to suit your own needs so you don't have to start from scratch.
Take a look at Performance Measurement Cookbook to see what other recipes Red Hat has to offer.