23.10. Using the Journal

The Journal is a component of systemd that is responsible for viewing and management of log files. It can be used in parallel, or in place of a traditional syslog daemon, such as rsyslogd. The Journal was developed to address problems connected with traditional logging. It is closely integrated with the rest of the system, supports various logging technologies and access management for the log files.
Logging data is collected, stored, and processed by the Journal's journald service. It creates and maintains binary files called journals based on logging information that is received from the kernel, from user processes, from standard output, and standard error output of system services or via its native API. These journals are structured and indexed, which provides relatively fast seek times. Journal entries can carry a unique identifier. The journald service collects numerous meta data fields for each log message. The actual journal files are secured, and therefore cannot be manually edited.

23.10.1. Viewing Log Files

To access the journal logs, use the journalctl tool. For a basic view of the logs type as root: 
journalctl
An output of this command is a list of all log files generated on the system including messages generated by system components and by users. The structure of this output is similar to one used in /var/log/messages/ but with certain improvements:
  • the priority of entries is marked visually. Lines of error priority and higher are highlighted with red color and a bold font is used for lines with notice and warning priority
  • the time stamps are converted for the local time zone of your system
  • all logged data is shown, including rotated logs
  • the beginning of a boot is tagged with a special line

Example 23.18. Example Output of journalctl

The following is an example output provided by the journalctl tool. When called without parameters, the listed entries begin with a time stamp, then the host name and application that performed the operation is mentioned followed by the actual message. This example shows the first three entries in the journal log: 
# journalctl
-- Logs begin at Thu 2013-08-01 15:42:12 CEST, end at Thu 2013-08-01 15:48:48 CEST. --
Aug 01 15:42:12 localhost systemd-journal[54]: Allowing runtime journal files to grow to 49.7M.
Aug 01 15:42:12 localhost kernel: Initializing cgroup subsys cpuset
Aug 01 15:42:12 localhost kernel: Initializing cgroup subsys cpu

[...]
In many cases, only the latest entries in the journal log are relevant. The simplest way to reduce journalctl output is to use the -n option that lists only the specified number of most recent log entries: 
journalctl -n Number
Replace Number with the number of lines to be shown. When no number is specified, journalctl displays the ten most recent entries.
The journalctl command allows controlling the form of the output with the following syntax: 
journalctl -o form
Replace form with a keyword specifying a desired form of output. There are several options, such as verbose, which returns full-structured entry items with all fields, export, which creates a binary stream suitable for backups and network transfer, and json, which formats entries as JSON data structures. For the full list of keywords, see the journalctl(1) manual page.

Example 23.19. Verbose journalctl Output

To view full meta data about all entries, type: 
# journalctl -o verbose
[...]

Fri 2013-08-02 14:41:22 CEST [s=e1021ca1b81e4fc688fad6a3ea21d35b;i=55c;b=78c81449c920439da57da7bd5c56a770;m=27cc
        _BOOT_ID=78c81449c920439da57da7bd5c56a770
        PRIORITY=5
        SYSLOG_FACILITY=3
        _TRANSPORT=syslog
        _MACHINE_ID=69d27b356a94476da859461d3a3bc6fd
        _HOSTNAME=localhost.localdomain
        _PID=562
        _COMM=dbus-daemon
        _EXE=/usr/bin/dbus-daemon
        _CMDLINE=/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
        _SYSTEMD_CGROUP=/system/dbus.service
        _SYSTEMD_UNIT=dbus.service
        SYSLOG_IDENTIFIER=dbus
        SYSLOG_PID=562
        _UID=81
        _GID=81
        _SELINUX_CONTEXT=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023
        MESSAGE=[system] Successfully activated service 'net.reactivated.Fprint'
        _SOURCE_REALTIME_TIMESTAMP=1375447282839181
        
[...]
This example lists fields that identify a single log entry. These meta data can be used for message filtering as shown in the section called “Advanced Filtering”. For a complete description of all possible fields see the systemd.journal-fields(7) manual page.

23.10.2. Access Control

By default, Journal users without root privileges can only see log files generated by them. The system administrator can add selected users to the adm group, which grants them access to complete log files. To do so, type as root: 
usermod -a -G adm username
Here, replace username with a name of the user to be added to the adm group. This user then receives the same output of the journalctl command as the root user. Note that access control only works when persistent storage is enabled for Journal.

23.10.3. Using The Live View

When called without parameters, journalctl shows the full list of entries, starting with the oldest entry collected. With the live view, you can supervise the log messages in real time as new entries are continuously printed as they appear. To start journalctl in live view mode, type: 
journalctl -f
This command returns a list of the ten most current log lines. The journalctl utility then stays running and waits for new changes to show them immediately.

23.10.4. Filtering Messages

The output of the journalctl command executed without parameters is often extensive, therefore you can use various filtering methods to extract information to meet your needs.

Filtering by Priority

Log messages are often used to track erroneous behavior on the system. To view only entries with a selected or higher priority, use the following syntax: 
journalctl -p priority
Here, replace priority with one of the following keywords (or with a number): debug (7), info (6), notice (5), warning (4), err (3), crit (2), alert (1), and emerg (0).

Example 23.20. Filtering by Priority

To view only entries with error or higher priority, use: 
journalctl -p err

Filtering by Time

To view log entries only from the current boot, type: 
journalctl -b
If you reboot your system just occasionally, the -b will not significantly reduce the output of journalctl. In such cases, time-based filtering is more helpful: 
journalctl --since=value --until=value
With --since and --until, you can view only log messages created within a specified time range. You can pass values to these options in form of date or time or both as shown in the following example.

Example 23.21. Filtering by Time and Priority

Filtering options can be combined to reduce the set of results according to specific requests. For example, to view the warning or higher priority messages from a certain point in time, type: 
journalctl -p warning --since="2013-3-16 23:59:59"

Advanced Filtering

Example 23.19, “Verbose journalctl Output” lists a set of fields that specify a log entry and can all be used for filtering. For a complete description of meta data that systemd can store, see the systemd.journal-fields(7) manual page. This meta data is collected for each log message, without user intervention. Values are usually text-based, but can take binary and large values; fields can have multiple values assigned though it is not very common.
To view a list of unique values that occur in a specified field, use the following syntax: 
journalctl -F fieldname
Replace fieldname with a name of a field you are interested in.
To show only log entries that fit a specific condition, use the following syntax: 
journalctl fieldname=value
Replace fieldname with a name of a field and value with a specific value contained in that field. As a result, only lines that match this condition are returned.

Note

As the number of meta data fields stored by systemd is quite large, it is easy to forget the exact name of the field of interest. When unsure, type: 
journalctl
and press the Tab key two times. This shows a list of available field names. Tab completion based on context works on field names, so you can type a distinctive set of letters from a field name and then press Tab to complete the name automatically. Similarly, you can list unique values from a field. Type: 
journalctl fieldname=
and press Tab two times. This serves as an alternative to journalctl -F fieldname.
You can specify multiple values for one field: 
journalctl fieldname=value1 fieldname=value2 ...
Specifying two matches for the same field results in a logical OR combination of the matches. Entries matching value1 or value2 are displayed.
Also, you can specify multiple field-value pairs to further reduce the output set: 
journalctl fieldname1=value fieldname2=value ...
If two matches for different field names are specified, they will be combined with a logical AND. Entries have to match both conditions to be shown.
With use of the + symbol, you can set a logical OR combination of matches for multiple fields: 
journalctl fieldname1=value + fieldname2=value ...
This command returns entries that match at least one of the conditions, not only those that match both of them.

Example 23.22. Advanced filtering

To display entries created by avahi-daemon.service or crond.service under user with UID 70, use the following command: 
journalctl _UID=70 _SYSTEMD_UNIT=avahi-daemon.service _SYSTEMD_UNIT=crond.service
Since there are two values set for the _SYSTEMD_UNIT field, both results will be displayed, but only when matching the _UID=70 condition. This can be expressed simply as: (UID=70 and (avahi or cron)).
You can apply the aforementioned filtering also in the live-view mode to keep track of the latest changes in the selected group of log entries: 
journalctl -f fieldname=value ...

23.10.5. Enabling Persistent Storage

By default, Journal stores log files only in memory or a small ring-buffer in the /run/log/journal/ directory. This is sufficient to show recent log history with journalctl. This directory is volatile, log data is not saved permanently. With the default configuration, syslog reads the journal logs and stores them in the /var/log/ directory. With persistent logging enabled, journal files are stored in /var/log/journal which means they persist after reboot. Journal can then replace rsyslog for some users (but see the chapter introduction).
Enabled persistent storage has the following advantages
  • Richer data is recorded for troubleshooting in a longer period of time
  • For immediate troubleshooting, richer data is available after a reboot
  • Server console currently reads data from journal, not log files
Persistent storage has also certain disadvantages:
  • Even with persistent storage the amount of data stored depends on free memory, there is no guarantee to cover a specific time span
  • More disk space is needed for logs
To enable persistent storage for Journal, create the journal directory manually as shown in the following example. As root type: 
mkdir -p /var/log/journal/
Then, restart journald to apply the change: 
systemctl restart systemd-journald