How to use ftrace in a vmcore file.
Environment
- Red Hat Enterprise Linux 6
Issue
- Is it possible to collect ftrace data in a vmcore?
- How to trace the ftrace data in crash
Resolution
-
If ftrace is active, the data also will be in a vmcore file.
-
To check the ftrace output with crash tool, you need to install crash-trace-command package.
$ yum install crash-trace-command
...
$ rpm -ql crash-trace-command
/usr/lib64/crash/extensions/trace.so
/usr/share/doc/crash-trace-command-1.0
/usr/share/doc/crash-trace-command-1.0/COPYING
- You need to load trace.so shared object in crash prompt as shown in below.
crash> extend /usr/lib64/crash/extensions/trace.so
/usr/lib64/crash/extensions/trace.so: shared object loaded
crash> help trace
NAME
trace - show or dump the tracing info
SYNOPSIS
trace [ <show [-c <cpulist>] [-f [no]<flagname>]> | <dump [-sm] <dest-dir>> ]
DESCRIPTION
trace
shows the current tracer and other informations.
trace show [ -c <cpulist> ] [ -f [no]<flagname> ]
shows all events with readability text(sorted by timestamp)
-c: only shows specified CPUs' events.
ex. trace show -c 1,2 - only shows cpu#1 and cpu#2 's events.
trace show -c 0,2-7 - only shows cpu#0, cpu#2...cpu#7's events.
-f: set or clear a flag
available flags default
context_info true
sym_offset false
sym_addr false
graph_print_duration true
graph_print_overhead true
graph_print_abstime false
graph_print_cpu true
graph_print_proc false
graph_print_overrun false
trace dump [-sm] <dest-dir>
dump ring_buffers to dest-dir. Then you can parse it
by other tracing tools. The dirs and files are generated
the same as debugfs/tracing.
-m: also dump metadata of ftrace.
-s: also dump symbols of the kernel <not implemented>.
crash> trace show
rtkit-daemon-2572 [001] 275386.148300: prev_pid=2572, prev_prio=120, prev_state=0, next_pid=2573, next_prio=0, next_state=0, next_cpu=0
rtkit-daemon-2572 [001] 275386.148304: default_wake_function <-pollwake
<swapper>-0 [000] 275386.148440: smp_reschedule_interrupt <-reschedule_interrupt
<swapper>-0 [000] 275386.148443: native_apic_mem_write <-smp_reschedule_interrupt
...
- You also can dump the ftrace ring buffer content into the directory for the future use. But, there isn't a good tool to analyse it at this stage.
crash> trace dump mydump
crash> !ls -l
total 27652
drwxr-xr-x. 3 root root 4096 Oct 25 15:24 mydump
-rw-------. 1 root root 28302914 Oct 25 12:30 vmcore
crash> sym -l > mydump/kallsyms
- RHEL6 ships trace-cmd as a command line tool to collect and report the output. But unfortunately, it's missing one feature (restore). So, if you want to see the output of the dumped raw file, you need to use upstream trace-cmd by pulling it from the git. I couldn't make it compile it in RHEL6, but it works fine in Fedora 17.
$ ls
mydump vmcore
$ sed -ne 's/(\(.*\))/\1/p' < mydump/kallsyms > mydump/functions
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
$ cd trace-cmd ; make gui
$ cd ..
$ ./trace-cmd/trace-cmd restore -c -k mydump/functions -o head.dat
$ ./trace-cmd/trace-cmd restore -i head.dat -o trace.dat mydump/per_cpu/cpu*/trace_pipe_raw
$ ./trace-cmd/trace-cmd report
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
