How to collect core dump file of a crashing program that is shipped in Red Hat Enterprise Linux
Environment
- Red Hat Enterprise Linux 6
- Red Hat Enterprise Linux 7
- Red Hat Enterprise Linux 8
- Red Hat Enterprise Linux 9
abrt
Issue
- How to collect core dump file of a program on RHEL6 and above
- How to report an application crash in RHEL6 and above
- How to configure abrt to automatically generate an application core
- How do I collect process core file in a container?
Resolution
-
Red Hat Enterprise Linux 6/7/8 has a service called
abrt
, which automatically collects the core dump files on the server and stores them inside the/var/spool/abrt
. -
ABRT is not present in RHEL 9, The Coredumps can be captured through systemd coredumpctl functionality.
-
Ensure that the system is set up to generate application cores by removing the core limits:
-
For the current interactive shell.
Check the current core ulimit.
# ulimit -c 0
Set the core ulimit to unlimited.
# ulimit -c unlimited
-
For a given user.
Login as the user and check the current core ulimit as above.
Edit /etc/security/limits.conf; see 'How to set ulimit values' for details.
-
For a RHEL 6 service.
Check the current core ulimit. Replace PID with the PID of your service.
# cat /proc/31220/limits Limit Soft Limit Hard Limit Units [snip] Max core file size 0 unlimited bytes [snip]
Add the desired ulimit command(s) to /etc/sysconfig/SERVICENAME; see 'How to set limits (ulimit) for services in RHEL6 and older with upstart, sysvinit' for details.
-
For a RHEL 7+ service.
Check the current core ulimit.
# systemctl show rsyslog | grep LimitCORE LimitCORE=0
Set the core ulimit to unlimited. Override the current ulimit by running,
# systemctl edit SERVICENAME
In the editor, add 'LimitCORE=infinity', example below.
[Service] LimitCORE=infinity
See 'How to set limits for services in RHEL and systemd' for more details.
-
-
Make sure abrt and abrt-addon-ccpp is installed
# yum install abrt abrt-addon-ccpp abrt-tui
-
Ensure the ccpp hooks are installed:
# abrt-install-ccpp-hook install # abrt-install-ccpp-hook is-installed; echo $?;
The second command should return 0 (hooks installed)
-
Ensure that this service is up and the ccpp hook to capture core dumps is enabled:
for RHEL6
# service abrtd start # service abrt-ccpp start
for RHEL6, configure services to start on boot. Optional
# chkconfig abrtd on # chkconfig abrt-ccpp on
for RHEL 7 or RHEL 8
# systemctl start abrtd.service # systemctl start abrt-ccpp.service
For RHEL 7 or RHEL 8, configure services to start on boot. Optional
# systemctl enable abrtd.service # systemctl enable abrt-ccpp.service
Additionally,
abrt-journal-core.service
might need to be disabled since it conflicts withabrt-ccpp.service
:# systemctl disable abrt-journal-core.service
-
To look for core dumps using the GUI, run the command:
for RHEL6
# abrt-gui
for RHEL7 or RHEL 8
# gnome-abrt
-
To get a list of crashes on the command line, issue the following command:
# abrt-cli list
-
To generate a report of a specific crash:
# abrt-cli report DUMPDIR
-
Choose the RHTSupport option to reports crashes into the Red Hat Technical Support system. Alternatively, you may use the abrt-cli command to get detailed information about particular crash:
# abrt-cli info DUMPDIR
-
Locate the coredump file path, and provide the entire directory to the Red Hat Support representative.
Show information on the DUMPDIR.
# abrt-cli info DUMPDIR id 68fb7093b18414f689d9c7bea9b12f55ea6822ba reason: rsyslogd killed by SIGBUS [snip] Directory: /var/spool/abrt/ccpp-2023-08-09-19:25:03-22545
Copy the DUMPDIR
# tar -cJf ~/rsyslog-crash-`hostname -s`.tar.xz /var/spool/abrt/ccpp-2023-08-09-19:25:03-22545
For more details refer to the following documents.
- RHEL6: Deployment GuideChapter 28. Automatic Bug Reporting Tool (ABRT)
-
RHEL7: System Administrator’s GuideChapter 25. Automatic Bug Reporting Tool (ABRT)
-
abrtd.service collect coredump file in a container.
Note for RHEL-6.1 and earlier:
The command invocations:
# abrt-cli list
# abrt-cli report
# abrt-cli info
were:
# abrt-cli --list
# abrt-cli --report
# abrt-cli --info
In RHEL versions 6.1 and earlier. Additionally, in RHEL-6.1 and earlier, abrt
refers to crashes by their UUID (obtained from the output of abrt-cli --list
) rather than the dump directory.
Enable dumping for unsigned packages
- To collect core dumps from unsigned packages, (such as third party or in-house developed applications), do the following:
- Edit
/etc/abrt/abrt-action-save-package-data.conf
- Change
OpenGPGCheck = yes
toOpenGPGCheck = no
- Save changes
- Edit
Enable core dumps for unpackaged software
- To collect core dumps from unpackaged software (such as in-house developed applications), do the following:
- Edit
/etc/abrt/abrt-action-save-package-data.conf
- Change
ProcessUnpackaged = no
toProcessUnpackaged = yes
- Save changes
- Edit
Diagnostic Steps
RHEL8 container
Run sleep command.
bash-4.4$ sleep 600 &
[1] 6
Check pid of sleep process.
bash-4.4$ ps aux | grep sleep
default 6 0.0 0.1 23044 1396 pts/0 S 06:06 0:00 /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 600
default 8 0.0 0.0 9204 1152 pts/0 S+ 06:07 0:00 grep sleep
Send SIGABRT signal to sleep process.
bash-4.4$ kill -SIGABRT 6
bash-4.4$
[1]+ Aborted (core dumped) sleep 600
bash-4.4$
Exit the container.
bash-4.4$ exit
exit
Run 'abrt-cli list'.
[root@podman83 ~]# abrt-cli list
id b633e788c46b56b1cae41db8bf4d17f49afbb315
reason: coreutils killed by SIGABRT
time: Mon May 31 15:11:05 2021
cmdline: /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 600
uid: 1001 ()
count: 1
Directory: /var/spool/abrt/ccpp-2021-05-31-15:11:05-77032
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments