8.8.3. Enabling Syslog for Cartridge Logs from Gears

By default, cartridge logs are written to the $OPENSHIFT_LOG_DIR directory of an application. You can configure logshifter on node hosts to instead have gears send their cartridge logs to Syslog. Starting with OpenShift Enterprise 2.1.7, you can also have them sent to both Syslog and the $OPENSHIFT_LOG_DIR directory at the same time.

Procedure 8.6. To Enable Syslog for Cartridge Logs from Gears:

  1. Edit the /etc/openshift/logshifter.conf file on the node host. The default value for the outputType setting is file, which results in gears sending cartridge logs to the $OPENSHIFT_LOG_DIR directory. Change this setting to syslog to have them sent to Syslog:
    outputType=syslog
    Alternatively, starting with OpenShift Enterprise 2.1.7, you can choose to change the outputType setting instead to multi, which results in logs being written using both file and syslog at the same time.
  2. Ask affected owners of existing applications to restart their applications for the changes to take effect. They can restart their applications with the following commands:
    $ rhc app stop -a appname
    $ rhc app start -a appname
    Alternatively, you can restart all gears on affected node hosts. The downtime caused by restarting all gears is minimal and normally lasts a few seconds:
    # oo-admin-ctl-gears restartall

Important

If the outputTypeFromEnviron setting in the /etc/openshift/logshifter.conf file is set to true, application owners are allowed to override the global outputType setting using a LOGSHIFTER_OUTPUT_TYPE environment variable in their application. See the OpenShift Enterprise User Guide for more information.
Cron Syslog Configuration

With the mmopenshift plug-in, all Cron cartridges will output all log information to the configured gear log file (/var/log/openshift_gears in the example 8.1). It may be necessary for system-level Cron logs to be separated from the gear logs for troubleshooting purposes. System-level Cron messages are tagged with cron_sys_log and can be separated into another file by adding the below to the /etc/syslog.conf Syslog configuration file:

:syslogtag, contains, "cron_sys_log:" /var/log/openshift_cron_cartridges.log
 &~

action(type="mmopenshift")

if $!OpenShift!OPENSHIFT_APP_UUID != '' then
  # annotate and log syslog output from gears specially
  *.*     action(type="omfile" file="/var/log/openshift_gears" template="OpenShift")
else
  # otherwise send syslog where it usually goes
  *.info;mail.none;authpriv.none;cron.none   action(type="omfile" file="/var/log/messages")
The :syslogtag entry must be placed before the *.* mmopenshift entry to prevent Cron system logs from going to both the openshift_cron_cartridges log and the openshift_gears log. The &~ tells Rsyslog to stop processing log entries if the filter condition on the previous line is met.
Enabling Application and Gear Context for Cartridge Logs

To provide context to cartridge logs aggregated to Syslog, a message modification plug-in for Rsyslog called mmopenshift can be used to add gear metadata to the cartridge logs. The plug-in can be configured to add metadata items to the JSON properties of each message that Rsyslog receives from a gear.

Due to configuration file format requirements, a newer version of Rsyslog, provided by the rsyslog7 package, must be installed and configured to enable this feature. The mmopenshift plug-in also only works for messages that have the $!uid JSON property, which can be added automatically when the imuxsock plug-in is enabled with the following options:
  • SysSock.Annotate
  • SysSock.ParseTrusted
  • SysSock.UsePIDFromSystem

Procedure 8.7. To Enable Application and Gear Context for Cartridge Logs:

  1. Install the mmopenshift plug-in, which requires the rsyslog7 package, on the node host. Because installing the rsyslog7 package where the rsyslog package is already installed can cause conflicts, consult the following instructions relevant to your node host.
    If the rsyslog package is already installed, use a yum shell to remove the rsyslog package and install the rsyslog7 and rsyslog7-mmopenshift packages safely:
    1. Stop the Rsyslog service:
      # service rsyslog stop
    2. Open a yum shell:
      # yum shell
    3. Run the following commands inside of the yum shell:
      > erase rsyslog
      > install rsyslog7 rsyslog7-mmopenshift
      > transaction run
      > quit
      The rsyslog package is uninstalled and a newer version of Rsyslog takes its place. The rsyslog7-mmopenshift package is also installed, which provides the mmopenshift module.
    Alternatively, if the rsyslog package is not already installed, or if rsyslog7 is already the only version of Rsyslog installed, install the mmopenshift module using the following command:
    # yum install rsyslog7 rsyslog7-mmopenshift
  2. Review the existing /etc/rsyslog.conf file, if relevant, and note any important default or custom settings. This includes changes that were made with the instructions described in Section 8.8.2, “Enabling Syslog for Node Components”. Next, make any required changes to ensure that the new /etc/rsyslog7.conf file contains those changes. Note that some settings may be different between /etc/rsyslog.conf and /etc/rsyslog7.conf.rpmnew; see http://www.rsyslog.com/doc/v7-stable/ for more information.Once complete, take a backup of /etc/rsyslog.conf and move /etc/rsyslog.conf.rpmnew to /etc/rsyslog.conf

    Important

    A sample section of an /etc/rsyslog7.conf.rpmnew file is provided at Example 8.8, “Sample Configuration Settings in /etc/rsyslog7.conf which depicts how the mmopenshift plug-in can be enabled for Rsyslog. However, it is not meant to represent a comprehensive /etc/rsyslog7.conf file or be fully comparable to the standard /etc/rsyslog.conf configuration.
  3. Edit the /etc/rsyslog7.conf file and add the following lines under the MODULES section to enable the imuxsock plug-in and the mmopenshift plug-in:
    module(load="imuxsock" SysSock.Annotate="on" SysSock.ParseTrusted="on" SysSock.UsePIDFromSystem="on")
    module(load="mmopenshift")
    
  4. Edit the /etc/rsyslog7.conf file and comment out the following line under the MODULES section to configure the imuxsock plug-in:
    #$ModLoad imuxsock
  5. Edit the /etc/rsyslog7.conf file and comment out the following lines to disable the imjournal plug-in:
    $ModLoad imjournal
    $OmitLocalLogging on
    $IMJournalStateFile imjournal.state
    
  6. Edit the /etc/rsyslog7.conf file to have Syslog search the /etc/rsyslog7.d directory for configuration files:
    #$IncludeConfig /etc/rsyslog.d/*.conf
    $IncludeConfig /etc/rsyslog7.d/*.conf
  7. Examine the /etc/rsyslog.d directory and copy any configuration files that are needed in /etc/rsyslog7.d directory for the Rsyslog7 logging configuration.
  8. Create a gear log template file in the Rsyslog7 directory. This defines the format of the gear logs, including sufficient parameters to distinguish gears from each other. This example template can be modified to suit the requirements of your log analysis tools. For more information on template configuration instructions, see http://www.rsyslog.com/doc/v7-stable/configuration/templates.html.:
    # vi /etc/rsyslog7.d/openshift-gear-template.conf
    template(name="OpenShift" type="list") {
            property(name="timestamp" dateFormat="rfc3339")
            constant(value=" ")
            property(name="hostname")
            constant(value=" ")
            property(name="syslogtag")
            constant(value=" app=")
            property(name="$!OpenShift!OPENSHIFT_APP_NAME")
            constant(value=" ns=")
            property(name="$!OpenShift!OPENSHIFT_NAMESPACE")
            constant(value=" appUuid=")
            property(name="$!OpenShift!OPENSHIFT_APP_UUID")
            constant(value=" gearUuid=")
            property(name="$!OpenShift!OPENSHIFT_GEAR_UUID")
            property(name="msg" spifno1stsp="on")
            property(name="msg" droplastlf="on")
            constant(value="\n")
    }
    
  9. Add the following lines to the /etc/rsyslog7.conf file under the RULES section to configure the mmopenshift plug-in to use the template from the previous step. The following example logs all gear messages to the /var/log/openshift_gears file and all other messages to the /var/log/messages file, but these destinations are configurable to a different destination:
    module(load="mmopenshift")
    action(type="mmopenshift")
    if $!OpenShift!OPENSHIFT_APP_UUID != '' then
      *.* action(type="omfile" file="/var/log/openshift_gears" template="OpenShift")
    else {
      *.info;mail.none;authpriv.none;cron.none   action(type="omfile" file="/var/log/messages")
    }
    Also, comment out the following line:
    # *.info;mail.none;authpriv.none;cron.none                /var/log/messages
  10. Start or restart the rsyslog service and ensure it starts persistently across reboots:
    # service rsyslog restart
    # chkconfig rsyslog on

Example 8.8. Sample Configuration Settings in /etc/rsyslog7.conf

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
#$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
#$ModLoad imjournal # provides access to the systemd journal
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability
module(load="imuxsock" SysSock.Annotate="on" SysSock.ParseTrusted="on" SysSock.UsePIDFromSystem="on")
module(load="mmopenshift")

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514

#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog7.d/
#$IncludeConfig /etc/rsyslog.d/*.conf
$IncludeConfig /etc/rsyslog7.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
#$OmitLocalLogging on

# File to store the position in the journal
#$IMJournalStateFile imjournal.state

#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
#*.info;mail.none;authpriv.none;cron.none                /var/log/messages
action(type="mmopenshift")
if $!OpenShift!OPENSHIFT_APP_UUID != '' then
  # annotate and log syslog output from gears specially
  *.*     action(type="omfile" file="/var/log/openshift_gears" template="OpenShift")
else
  # otherwise send syslog where it usually goes
  *.info;mail.none;authpriv.none;cron.none   action(type="omfile" file="/var/log/messages")
  
# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                               /var/log/boot.log