How to enable persistent logging for the systemd journal

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 9
  • Red Hat Enterprise Linux 10

Issue

  • The systemd journal is configured by default to store logs only in a small ring-buffer in /run/log/journal, which is not persistent.

  • Journal database logs do not survive a system reboot.

Resolution

  • There are two methods to enable persistent systemd journal logs in /var/log/journal.

Method 1

  • Create the directory journal using the below command:

    # mkdir -p /var/log/journal

  • Once the above directory is created the system will automatically start storing the logs at /var/log/journal once any loggable event is encountered in RHEL 7 & RHEL 8.

  • After that run the below command only in RHEL 9 and RHEL 10:

    # journalctl --flush

Method 2

Step 1. Edit Journald configuration

  • For RHEL 9 and below set the Storage parameter equals to persistent in /etc/systemd/journald.conf:

    # sed -i 's/#Storage=auto/Storage=persistent/' /etc/systemd/journald.conf

  • In RHEL 10, location of main configuration files is changed to /usr/lib/systemd/ folder. Still in /etc/systemd/ folder override configurations can be placed like this.

    # cat /etc/systemd/journald.conf
[Journal]
Storage=persistent

  • With the storage option persistent, data will be stored preferably on disk, i.e. below the /var/log/journal hierarchy (which is created if needed), with a fallback to /run/log/journal (which is created if needed), during early boot and if the disk is not writable.

Step 2 Varies depending on the RHEL version:

  • On RHEL 7 & 8, restart the systemd-journald.service:

    # systemctl restart systemd-journald.service

  • On RHEL 9 and RHEL 10:

    # systemctl restart systemd-journald.service
# journalctl --flush

  • Note: In RHEL 9, journalctl --flush is required after restarting the systemd-journald.service. This makes the journal work nicer with systems which have the root file system writable early, but still need to rearrange /var before journald should start writing and creating files to it, for example because ACLs need to be applied first, or because /var is to be mounted from another file system, NFS or tmpfs.

Additional Information

Root Cause

  • By default in Red Hat Enterprise Linux 7, 8, 9 & 10 systemd is not configured to maintain logs persistently.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments