How do I configure logging for pacemaker in RHEL 6, 7, 8 or 9 High Availability clusters?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 6, 7, 8 or 9 with the High Availability Add On
  • pacemaker

Issue

  • Where does pacemaker log to?
  • Can I set where individual pacemaker-related daemons like crmd, lrmd, stonith, and others log to individually and independent of each other?
  • How can I change the logging level for pacemaker related daemons?

Resolution

General Configuration

pacemaker-related logging can be configured through /etc/sysconfig/pacemaker.

NOTE: A bug in certain releases of pacemaker in RHEL 6 prevents these settings from /etc/sysconfig/pacemaker from being honored .

On each node that has had its logging settings changed:

# pcs cluster stop
# pcs cluster start

Default "info" Level File Logging:

All messages from pacemaker-related daemons of priority INFO and higher will be logged directly to a file. The location of this file can be specified via PCMK_debugfile in RHEL 7, and PCMK_logfile in RHEL 8+:

  • NOTE: Setting The logfile setting in corosync.conf will override this setting for RHEL 7, if another value is not specified in /etc/sysconfig/pacemaker as well. This value is unspecified in corosync.conf by default, leaving this default setting of PCMK_debugfile=/var/log/pacemaker.log to take effect.

  • If the preference is to stop all "info" and higher level logging to /var/log/pacemaker/pacemaker.log, then this value can be set to none.

$ vim /etc/sysconfig/pacemaker
-----------------------------------------8<----------------------------------------- 
# RHEL 7:
-----------------------------------------8<----------------------------------------- 
# Send INFO (and higher) messages to the named log file
# Additional messages may also appear here depending on any configured debug and trace settings
# By default Pacemaker will inherit the logfile specified in corosync.conf
PCMK_debugfile=/var/log/pacemaker.log 

$ vim /etc/sysconfig/pacemaker
-----------------------------------------8<----------------------------------------- 
# RHEL 8:
# PCMK_logfile
# Unless set to "none", more detailed log messages will be sent to the
# specified file (in addition to the system log, if enabled). These messages
# may have extended information, and will include messages of info severity.
# This log is of more use to developers and advanced system administrators, and
# when reporting problems.
#
# Default: PCMK_logfile="/var/log/pacemaker/pacemaker.log"
PCMK_logfile=/var/log/pacemaker/pacemaker.log

In RHEL 6, this logfile value comes from the corosync logging_daemon logfile setting in /etc/cluster/cluster.conf, and defaults to /var/log/cluster/corosync.log:

<logging>
   <logging_daemon name="corosync" logfile="/var/log/cluster/corosync.log"/>
</logging>

Enabling debug logging:

To enable debug logging to this file for any specific daemon, or for all or none, use PCMK_debug in /etc/sysconfig/pacemaker:

# Enable debug logging globally or per-subsystem
# Multiple subsystems may me listed separated by commas
# eg. PCMK_debug=crmd,pengine
# PCMK_debug=yes|no|crmd|pengine|cib|stonith-ng|attrd|pacemakerd

NOTE: Enabling logging for all daemons with "yes" can produce significant amounts of data being written to disk, and the preferred value should come from the specific circumstances:

  • PCMK_debug=yes is typically preferred in cases where an issue that needs debugging or diagnosing can be reproduced at-will, or with only a short amount of waiting. This will ensure all relevant events are logged and available for review by Red Hat Global Support Services or other advanced users.

  • In other scenarios it may be preferable to limit debug logging to specific daemons while waiting for an issue, or set to no for general use cases that do not require constant debug logging.

Syslog Logging

pacemaker-related daemons will have certain messages of a specified priority or higher automatically propagated to syslog, whereas others that may be informational only or debug-related will only go to a separate file ( specified by PCMK_logfile ). To control the priority at which messages will go to syslog in addition to the file, use PCMK_logpriority, which defaults to notice:

# Send all messages up-to-and-including the configured priority to syslog
# A value of 'info' will be far too verbose for most installations and 'debug' is almost certain to send you blind
# The default value is 'notice'
# PCMK_logpriority=emerg|alert|crit|error|warning|notice|info|debug

To control the syslog facility these messages will carry, use PCMK_logfacility, which defaults to daemon:

# Specify an alternate syslog target for NOTICE (and higher) messages
# Use 'none' to disable - not recommended
# The default value is 'daemon'
# PCMK_logfacility=none|daemon|user|local0|local1|local2|local3|local4|local5|local6|local7

NOTE: Messages of the default facility and priority should be directed to /var/log/messages. If the above settings are changed, additional policies may need to be implemented in /etc/rsyslog.conf to direct these messages to the desired location.

Log events from different processes to a different file

The service rsyslog can be configured to write events from different processes to different files. For example, we can write all events related to pacemaker to /var/log/pacemaker.log except for pacemakerd which will use another file to the events for that process (/var/log/pacemakerd.log) :

Add the following into your /etc/rsyslog.conf then restart rsyslog.

:app-name, startswith, "pacemakerd" /var/log/pacemakerd.log 
:app-name, startswith, "attrd" /var/log/pacemaker.log 
:app-name, startswith, "crmd" /var/log/pacemaker.log 
:app-name, startswith, "pengine" /var/log/pacemaker.log 
:app-name, startswith, "cib" /var/log/pacemaker.log 
:app-name, startswith, "stonith-ng" /var/log/pacemaker.log 
:app-name, startswith, "pacemaker" /var/log/pacemaker.log

Root Cause

pacemaker-related daemons share a common logging infrastructure that is set up by pacemakerd to use libqb and syslog. Messages of importance should go to both a file and to syslog (usually /var/log/messages), whereas more informational or debug messages will go to a file. There is no method to split out individual daemons to different logging facilities or files with pacemaker; they must all go to one location.

The default file that is used for logging (other than /var/log/messages) is the one that is set by the corosync process which is usually /var/log/cluster/corosync.conf.

# grep PCMK_logfile /etc/sysconfig/pacemaker -B 1
# By default Pacemaker will inherit the logfile specified in corosync.conf
# PCMK_logfile=/var/log/pacemaker.log

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments