How to collect core dump file of a crashing program that is shipped in Red Hat Enterprise Linux 6/7/8?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 8
  • abrt

Issue

  • How to collect core dump file of a program on RHEL6 and above
  • How to report an application crash in RHEL6 and above
  • How to configure abrt to automatically generate an application core

Resolution

Red Hat Enterprise Linux 6/7/8 has a service called abrt, which automatically collects the core dump files on the server and stores them inside the /var/spool/abrt.

  • Ensure that the system is set up to generate application cores by removing the core limits:
# ulimit -c unlimited
  • Make sure abrt and abrt-addon-ccpp is installed
# yum install abrt abrt-addon-ccpp abrt-tui
  • Ensure the ccpp hooks are installed:
# abrt-install-ccpp-hook install
# abrt-install-ccpp-hook is-installed; echo $?;
  • The second command should return 0 (hooks installed)

  • Ensure that this service is up and the ccpp hook to capture core dumps is enabled:

# service abrtd start
# service abrt-ccpp start

for RHEL 6 and

# systemctl start abrtd.service
# systemctl start abrt-ccpp.service

for RHEL 7/8.

  • To look for core dumps using the GUI, run the command:
# abrt-gui
  • To get a list of crashes on the command line, issue the following command:
# abrt-cli list
  • To generate a report of a specific crash:
# abrt-cli report DUMPDIR
  • Choose the RHTSupport option to reports crashes into the Red Hat Technical Support system. Alternatively, you may use the abrt-cli command to get detailed information about particular crash:
# abrt-cli info DUMPDIR
  • Locate the coredump file path, and provide the entire directory to the Red Hat Support representative.

For more details refer to the Red Hat Enterprise Linux Deployment Guide

Note for RHEL-6.1 and earlier:

The command invocations:

abrt-cli list
abrt-cli report
abrt-cli info

were:

abrt-cli --list
abrt-cli --report
abrt-cli --info

in RHEL versions 6.1 and earlier. Additionally in RHEL-6.1 and earlier, abrt refers to crashes by their UUID (obtained from the output of abrt-cli --list) rather than the dump directory.

Enable dumping for unsigned packages

  • To collect core dumps from unsigned packages, (such as third party or in-house developed applications), do the following:
    • Edit /etc/abrt/abrt-action-save-package-data.conf
    • Change OpenGPGCheck = yes to OpenGPGCheck = no
    • Save changes and restart the daemon

Enable core dumps for unpackaged software

  • To collect core dumps from unpackaged software (such as in-house developed applications), do the following:
    • Edit /etc/abrt/abrt-action-save-package-data.conf
    • Change ProcessUnpackaged = no to ProcessUnpackaged = yes
    • Save changes and restart the daemon

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

12 Comments

Also, ensure that abrtd and abrt-ccpp are permanently enabled

note that you do need to install the abrt and abrt-addon-ccpp packages

For RHEL 7 and systemd:
# ulimit -c unlimited
I doubt it will work at all, if you want to debug a daemon and the like, Ulimit only applies to child processes of a shell.
I have failed to enable core dumps on RHEL 7.1 following this solution.

Could you please correct that!

Thank you!

For RHEL 7, there seems to be one step missing, which is documented at the bottom of this page:

https://access.redhat.com/solutions/4896

With abrt employed, you don't even need to touch ulimit at all and abrt will capture all core dump files - daemon, user process, etc.. However, there are various circumstances in which abrt does not catch a core dump file because the core dump file is not produced - there will be no core dump file even without abrt and using 'ulimit -c unlimited'. See 'man 5 core' for more details.

The most common case where a core dump file is not produced is when a process is executing a set-user-ID (set-group-ID) program. If you need to have a core dump file for such a process you need to set the /proc/sys/fs/suid_dumpable file to 2.

Either:

echo 2 > /proc/sys/fs/suid_dumpable

or:

sysctl fs.suid_dumpable=2

Nice work.

it was so helpful. Now we configured on our environment.And how to read core dump file? using gdb or crash utility

Use gdb. First of all you need to have debuginfo available. If you open a core dump file in gdb, it should show you a command which you can use to install the missing debuginfo packages.

If you want to just see the backtrace without the need to delve into gdb details, you can use abrt to get the backtrace from the core dump file. Go to the relevant '/var/spool/abrt/ccpp-' directory and run abrt-action-analyze-ccpp-local which should donwload all required debuginfo data and create a new file named *backtrace containing the backtrace produced by gdb.

Thanks for your update.

On RHEL7 you need to install abrt-cli:

yum install abrt-cli -y

i have done configuration as per links, but logs are not getting generate.

rhel 6.5

Please modify the line, yum install abrt abrt-addon-ccpp abrt-tui to yum install abrt abrt-addon-ccpp abrt-gui