-
Language:
English
-
Language:
English
8.8.3. Enabling Syslog for Cartridge Logs from Gears
$OPENSHIFT_LOG_DIR
directory of an application. You can configure logshifter
on node hosts to instead have gears send their cartridge logs to Syslog. Starting with OpenShift Enterprise 2.1.7, you can also have them sent to both Syslog and the $OPENSHIFT_LOG_DIR
directory at the same time.
Procedure 8.6. To Enable Syslog for Cartridge Logs from Gears:
- Edit the
/etc/openshift/logshifter.conf
file on the node host. The default value for theoutputType
setting isfile
, which results in gears sending cartridge logs to the$OPENSHIFT_LOG_DIR
directory. Change this setting tosyslog
to have them sent to Syslog:outputType=syslog
Alternatively, starting with OpenShift Enterprise 2.1.7, you can choose to change theoutputType
setting instead tomulti
, which results in logs being written using bothfile
andsyslog
at the same time. - Ask affected owners of existing applications to restart their applications for the changes to take effect. They can restart their applications with the following commands:
$
rhc app stop -a appname
$rhc app start -a appname
Alternatively, you can restart all gears on affected node hosts. The downtime caused by restarting all gears is minimal and normally lasts a few seconds:#
oo-admin-ctl-gears restartall
Important
outputTypeFromEnviron
setting in the /etc/openshift/logshifter.conf
file is set to true
, application owners are allowed to override the global outputType
setting using a LOGSHIFTER_OUTPUT_TYPE
environment variable in their application. See the OpenShift Enterprise User Guide for more information.
With the mmopenshift
plug-in, all Cron cartridges will output all log information to the configured gear log file (/var/log/openshift_gears
in the example 8.1). It may be necessary for system-level Cron logs to be separated from the gear logs for troubleshooting purposes. System-level Cron messages are tagged with cron_sys_log
and can be separated into another file by adding the below to the /etc/syslog.conf
Syslog configuration file:
:syslogtag, contains, "cron_sys_log:" /var/log/openshift_cron_cartridges.log &~ action(type="mmopenshift") if $!OpenShift!OPENSHIFT_APP_UUID != '' then # annotate and log syslog output from gears specially *.* action(type="omfile" file="/var/log/openshift_gears" template="OpenShift") else # otherwise send syslog where it usually goes *.info;mail.none;authpriv.none;cron.none action(type="omfile" file="/var/log/messages")
:syslogtag
entry must be placed before the *.* mmopenshift
entry to prevent Cron system logs from going to both the openshift_cron_cartridges
log and the openshift_gears
log. The &~
tells Rsyslog to stop processing log entries if the filter condition on the previous line is met.
To provide context to cartridge logs aggregated to Syslog, a message modification plug-in for Rsyslog called mmopenshift
can be used to add gear metadata to the cartridge logs. The plug-in can be configured to add metadata items to the JSON properties of each message that Rsyslog receives from a gear.
mmopenshift
plug-in also only works for messages that have the $!uid
JSON property, which can be added automatically when the imuxsock
plug-in is enabled with the following options:
- SysSock.Annotate
- SysSock.ParseTrusted
- SysSock.UsePIDFromSystem
Procedure 8.7. To Enable Application and Gear Context for Cartridge Logs:
- Install the
mmopenshift
plug-in, which requires the rsyslog7 package, on the node host. Because installing the rsyslog7 package where the rsyslog package is already installed can cause conflicts, consult the following instructions relevant to your node host.If the rsyslog package is already installed, use ayum
shell to remove the rsyslog package and install the rsyslog7 and rsyslog7-mmopenshift packages safely:- Stop the Rsyslog service:
#
service rsyslog stop
- Open a
yum
shell:#
yum shell
- Run the following commands inside of the
yum
shell:> erase rsyslog
> install rsyslog7 rsyslog7-mmopenshift
> transaction run
> quit
The rsyslog package is uninstalled and a newer version of Rsyslog takes its place. The rsyslog7-mmopenshift package is also installed, which provides themmopenshift
module.
Alternatively, if the rsyslog package is not already installed, or if rsyslog7 is already the only version of Rsyslog installed, install themmopenshift
module using the following command:#
yum install rsyslog7 rsyslog7-mmopenshift
- Review the existing
/etc/rsyslog.conf
file, if relevant, and note any important default or custom settings. This includes changes that were made with the instructions described in Section 8.8.2, “Enabling Syslog for Node Components”. Next, make any required changes to ensure that the new/etc/rsyslog7.conf
file contains those changes. Note that some settings may be different between/etc/rsyslog.conf
and/etc/rsyslog7.conf.rpmnew
; see http://www.rsyslog.com/doc/v7-stable/ for more information.Once complete, take a backup of/etc/rsyslog.conf
and move/etc/rsyslog.conf.rpmnew
to/etc/rsyslog.conf
Important
A sample section of an/etc/rsyslog7.conf.rpmnew
file is provided at Example 8.8, “Sample Configuration Settings in/etc/rsyslog7.conf
” which depicts how themmopenshift
plug-in can be enabled for Rsyslog. However, it is not meant to represent a comprehensive/etc/rsyslog7.conf
file or be fully comparable to the standard/etc/rsyslog.conf
configuration. - Edit the
/etc/rsyslog7.conf
file and add the following lines under theMODULES
section to enable theimuxsock
plug-in and themmopenshift
plug-in:module(load="imuxsock" SysSock.Annotate="on" SysSock.ParseTrusted="on" SysSock.UsePIDFromSystem="on") module(load="mmopenshift")
- Edit the
/etc/rsyslog7.conf
file and comment out the following line under theMODULES
section to configure theimuxsock
plug-in:#$ModLoad imuxsock
- Edit the
/etc/rsyslog7.conf
file and comment out the following lines to disable theimjournal
plug-in:$ModLoad imjournal $OmitLocalLogging on $IMJournalStateFile imjournal.state
- Edit the
/etc/rsyslog7.conf
file to have Syslog search the/etc/rsyslog7.d
directory for configuration files:#$IncludeConfig /etc/rsyslog.d/*.conf $IncludeConfig /etc/rsyslog7.d/*.conf
- Examine the
/etc/rsyslog.d
directory and copy any configuration files that are needed in/etc/rsyslog7.d
directory for the Rsyslog7 logging configuration. - Create a gear log template file in the Rsyslog7 directory. This defines the format of the gear logs, including sufficient parameters to distinguish gears from each other. This example template can be modified to suit the requirements of your log analysis tools. For more information on template configuration instructions, see http://www.rsyslog.com/doc/v7-stable/configuration/templates.html.:
#
vi /etc/rsyslog7.d/openshift-gear-template.conf
template(name="OpenShift" type="list") { property(name="timestamp" dateFormat="rfc3339") constant(value=" ") property(name="hostname") constant(value=" ") property(name="syslogtag") constant(value=" app=") property(name="$!OpenShift!OPENSHIFT_APP_NAME") constant(value=" ns=") property(name="$!OpenShift!OPENSHIFT_NAMESPACE") constant(value=" appUuid=") property(name="$!OpenShift!OPENSHIFT_APP_UUID") constant(value=" gearUuid=") property(name="$!OpenShift!OPENSHIFT_GEAR_UUID") property(name="msg" spifno1stsp="on") property(name="msg" droplastlf="on") constant(value="\n") } - Add the following lines to the
/etc/rsyslog7.conf
file under theRULES
section to configure themmopenshift
plug-in to use the template from the previous step. The following example logs all gear messages to the/var/log/openshift_gears
file and all other messages to the/var/log/messages
file, but these destinations are configurable to a different destination:module(load="mmopenshift") action(type="mmopenshift") if $!OpenShift!OPENSHIFT_APP_UUID != '' then *.* action(type="omfile" file="/var/log/openshift_gears" template="OpenShift") else { *.info;mail.none;authpriv.none;cron.none action(type="omfile" file="/var/log/messages") }
Also, comment out the following line:# *.info;mail.none;authpriv.none;cron.none /var/log/messages
- Start or restart the
rsyslog
service and ensure it starts persistently across reboots:#
service rsyslog restart
#chkconfig rsyslog on
Example 8.8. Sample Configuration Settings in /etc/rsyslog7.conf
#### MODULES #### # The imjournal module bellow is now used as a message source instead of imuxsock. #$ModLoad imuxsock # provides support for local system logging (e.g. via logger command) #$ModLoad imjournal # provides access to the systemd journal $ModLoad imklog # provides kernel logging support (previously done by rklogd) #$ModLoad immark # provides --MARK-- message capability module(load="imuxsock" SysSock.Annotate="on" SysSock.ParseTrusted="on" SysSock.UsePIDFromSystem="on") module(load="mmopenshift") # Provides UDP syslog reception #$ModLoad imudp #$UDPServerRun 514 # Provides TCP syslog reception #$ModLoad imtcp #$InputTCPServerRun 514 #### GLOBAL DIRECTIVES #### # Where to place auxiliary files $WorkDirectory /var/lib/rsyslog # Use default timestamp format $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # File syncing capability is disabled by default. This feature is usually not required, # not useful and an extreme performance hit #$ActionFileEnableSync on # Include all config files in /etc/rsyslog7.d/ #$IncludeConfig /etc/rsyslog.d/*.conf $IncludeConfig /etc/rsyslog7.d/*.conf # Turn off message reception via local log socket; # local messages are retrieved through imjournal now. #$OmitLocalLogging on # File to store the position in the journal #$IMJournalStateFile imjournal.state #### RULES #### # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! #*.info;mail.none;authpriv.none;cron.none /var/log/messages action(type="mmopenshift") if $!OpenShift!OPENSHIFT_APP_UUID != '' then # annotate and log syslog output from gears specially *.* action(type="omfile" file="/var/log/openshift_gears" template="OpenShift") else # otherwise send syslog where it usually goes *.info;mail.none;authpriv.none;cron.none action(type="omfile" file="/var/log/messages") # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg :omusrmsg:* # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log