How to use ftrace in a vmcore file.

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 6

Issue

  • Is it possible to collect ftrace data in a vmcore?
  • How to trace the ftrace data in crash

Resolution

  • If ftrace is active, the data also will be in a vmcore file.

  • To check the ftrace output with crash tool, you need to install crash-trace-command package.

$ yum install crash-trace-command
...
$ rpm -ql crash-trace-command
/usr/lib64/crash/extensions/trace.so
/usr/share/doc/crash-trace-command-1.0
/usr/share/doc/crash-trace-command-1.0/COPYING
  • You need to load trace.so shared object in crash prompt as shown in below.
crash> extend /usr/lib64/crash/extensions/trace.so
/usr/lib64/crash/extensions/trace.so: shared object loaded
crash> help trace

NAME
  trace - show or dump the tracing info

SYNOPSIS
  trace [ <show [-c <cpulist>] [-f [no]<flagname>]> | <dump [-sm] <dest-dir>> ]

DESCRIPTION
trace
    shows the current tracer and other informations.

trace show [ -c <cpulist> ] [ -f [no]<flagname> ]
    shows all events with readability text(sorted by timestamp)
    -c: only shows specified CPUs' events.
        ex. trace show -c 1,2    - only shows cpu#1 and cpu#2 's events.
            trace show -c 0,2-7  - only shows cpu#0, cpu#2...cpu#7's events.
    -f: set or clear a flag
        available flags            default
        context_info               true
        sym_offset                 false
        sym_addr                   false
        graph_print_duration       true
        graph_print_overhead       true
        graph_print_abstime        false
        graph_print_cpu            true
        graph_print_proc           false
        graph_print_overrun        false

trace dump [-sm] <dest-dir>
    dump ring_buffers to dest-dir. Then you can parse it
    by other tracing tools. The dirs and files are generated
    the same as debugfs/tracing.
    -m: also dump metadata of ftrace.
    -s: also dump symbols of the kernel <not implemented>.

crash> trace show
    rtkit-daemon-2572  [001] 275386.148300: prev_pid=2572, prev_prio=120, prev_state=0, next_pid=2573, next_prio=0, next_state=0, next_cpu=0
    rtkit-daemon-2572  [001] 275386.148304: default_wake_function <-pollwake
       <swapper>-0     [000] 275386.148440: smp_reschedule_interrupt <-reschedule_interrupt
       <swapper>-0     [000] 275386.148443: native_apic_mem_write <-smp_reschedule_interrupt
...
  • You also can dump the ftrace ring buffer content into the directory for the future use. But, there isn't a good tool to analyse it at this stage.
crash> trace dump mydump
crash> !ls -l
total 27652
drwxr-xr-x. 3 root root     4096 Oct 25 15:24 mydump
-rw-------. 1 root root 28302914 Oct 25 12:30 vmcore
crash> sym -l > mydump/kallsyms
  • RHEL6 ships trace-cmd as a command line tool to collect and report the output. But unfortunately, it's missing one feature (restore). So, if you want to see the output of the dumped raw file, you need to use upstream trace-cmd by pulling it from the git. I couldn't make it compile it in RHEL6, but it works fine in Fedora 17.
$ ls
mydump vmcore
$ sed -ne 's/(\(.*\))/\1/p' < mydump/kallsyms > mydump/functions
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
$ cd trace-cmd ; make gui
$ cd ..
$ ./trace-cmd/trace-cmd restore -c -k mydump/functions -o head.dat
$ ./trace-cmd/trace-cmd restore -i head.dat -o trace.dat mydump/per_cpu/cpu*/trace_pipe_raw
$ ./trace-cmd/trace-cmd report

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.