7.5. Replacing Log Files with a Named Pipe

Many administrators want to do some special configuration or operation with logging data, like configuring an access log to record only certain events. This is not possible using the standard Directory Server log file configuration attributes, but it is possible by sending the log data to a named pipe, and then using another script to process the data. Using a named pipe for the log simplifies these special tasks, like:
  • Logging certain events, like failed bind attempts or connections from specific users or IP addresses
  • Logging entries which match a specific regular expression pattern
  • Keeping the log to a certain length (logging only the last number of lines)
  • Sending a notification, such as an email, when an event occurs
Replacing a log file with a pipe improves performance, especially on servers with a high rate of operations.
The named pipe is different than using a script to extract data from the logs because of how data are handled in the log buffer.
If a log is buffered, server performance is good, but important data are not written to disk (the log file) as soon as the event occurs. If the server is having a problem with crashing, it may crash before the data is written to disk — and there is no data for the script to extract.
If a log is not buffered[1], the writes are flushed to disk with each operation, causing a lot of disk I/O and performance degradation.
Replacing the log disk file with a pipe has the benefits of buffering, since the script that reads from the pipe can buffer the incoming log data in memory (which is not possible with a simple script).
The usage and option details for the script is covered in Section 9.4, “ds-logpipe.py”. The basic format is:

ds-logpipe.py /path/to/named_pipe [ --user pipe_user ] [ --maxlines number ] [[ --serverpidfile file.pid ] | [ --serverpid PID ]] [ --servertimeout seconds ] [ --plugin=/path/to/plugin.py | [ pluginfile.arg=value ]]

7.5.1. Using the Named Pipe for Logging

The Directory Server instance can use a named pipe for its logging simply by running the named pipe log script and giving the name of the pipe. (If the server is already running, then the log has to be reopened, but there is no configuration required otherwise.) 
# ds-logpipe.py /var/log/dirsrv/slapd-example/access
Running the ds-logpipe.py in this way has the advantage of being simple to implement and not requiring any Directory Server configuration changes. This is useful for fast debugging or monitoring, especially if you are looking for a specific type of event.
If the Directory Server instance will frequently or permanently use the named pipe rather than a real file for logging, then it is possible to reconfigure the instance to create the named pipe and use it for logging (as it does by default for the log files).
Three things need to be configured for the log configuration for the instance:
  • The log file to use has to be changed to the pipe (nsslapd-*log, where the * can be access, error, or audit[2], depending on the log type being configured)
  • Buffering should be disabled because the script already buffers the log entries (nsslapd-*log-logbuffering)
  • Log rotation should be disabled so that the server does not attempt to rotate the named pipe (nsslapd-*log-maxlogsperdir, nsslapd-*log-logexpirationtime, and nsslapd-*log-logrotationtime)
These configuration changes can be made in the Directory Server Console or using ldapmodify.
For example, this switches the access log to access.pipe: 
# ldapmodify -D "cn=Directory Manager" -W -p 389 -h server.example.com -x

dn: cn=config
changetype: modify
replace: nsslapd-accesslog
nsslapd-accesslog: /var/log/dirsrv/slapd-instance/access.pipe
-
replace: nsslapd-accesslog-logbuffering
nsslapd-accesslog-logbuffering: off
-
replace: nsslapd-accesslog-maxlogsperdir
nsslapd-accesslog-maxlogsperdir: 1
- 
replace: nsslapd-accesslog-logexpirationtime
nsslapd-accesslog-logexpirationtime: -1
- 
replace: nsslapd-accesslog-logrotationtime
nsslapd-accesslog-logrotationtime: -1

Note

Making these changes causes the server to close the current log file and switch to the named pipe immediately. This can be very helpful for debugging a running server and sifting the log output for specific messages.

7.5.2. Starting the Named Pipe with the Server

The named pipe can be started and shut down along with the Directory Server instance by editing the instance's init script configuration file.

Note

The named pipe script has to be specifically configured in the instance's dse.ldif file before it can be called at server startup.
  1. Open the instance configuration file for the server system. 
    /etc/sysconfig/dirsrv-instance_name

    Warning

    Do not edit the /etc/sysconfig/dirsrv file.
  2. At the end of the file, there will be a line that reads: 
    # Put custom instance specific settings below here.
    Below that line, insert the ds-logpipe.py command to launch when the server starts. For example: 
    # only keep the last 1000 lines of the error log
python /usr/bin/ds-logpipe.py /var/log/dirsrv/slapd-example/errors.pipe -m 1000 -u dirsrv -s /var/run/dirsrv/slapd-example.pid > /var/log/dirsrv/slapd-example/errors &

# only log failed binds
python /usr/bin/ds-logpipe.py /var/log/dirsrv/slapd-example/access.pipe -u dirsrv -s /var/run/dirsrv/slapd-example.pid --plugin=/usr/share/dirsrv/data/failedbinds.py failedbinds.logfile=/var/log/dirsrv/slapd-example/access.failedbinds &

    Note

    The -s option both specifies the .pid file for the server to write its PID to and sets the script to start and stop with the server process.

7.5.3. Using Plug-ins with the Named Pipe Log

A plug-in can be called to read the log data from the named pipe and perform some operation on it. There are some considerations with using plug-ins with the named pipe log script:
  • The plug-in function is called for every line read from the named pipe.
  • The plug-in function must be a Python script and must end in .py.
  • Any plug-in arguments are passed in the command line to the named pipe log script.
  • A pre-operation function can be specified for when the plug-in is loaded.
  • A post-operation function can be called for when the script exits.

7.5.3.1. Loading Plug-ins with the Named Pipe Log Script

There are two options with ds-logpipe.py to use for plug-ins:
  • The --plugin option gives the path to the plug-in file (which must be a Python script and must end in .py).
  • The plugin.arg option passes plug-in arguments to the named pipe log script. The plug-in file name (without the .py extension) is plugin and any argument allowed in that plug-in can be arg .
For example: 
ds-logpipe.py /var/log/dirsrc/slapd-example/errors.pipe --plugin=/usr/share/dirsrv/data/example-funct.py example-funct.regex="warning" > warnings.txt
If there are more than one values passed for the same argument, then they are converted into a list of values in the plug-in dict. For example, this script gives two values for arg1: 
--plugin=/path/to/pluginname.py pluginname.arg1=foo pluginname.arg1=bar pluginname.arg2=baz
In the plug-in, this is converted to: 
{'arg1': ['foo', 'bar'],
 'arg2': 'baz'}
This is a Python dict object with two keys. The first key is the string arg1, and its value is a Python list object with two elements, the strings foo and bar. The second key is the string arg2, and its value is the string baz. If an argument has only a single value, it is left as a simple string. Multiple values for a single argument name are converted into a list of strings.

7.5.3.2. Writing Plug-ins to Use with the Named Pipe Log Script

The ds-logpipe.py command expects up to three functions in any plug-in: plugin (), pre (), and post ().
Any plug-in used with the ds-logpipe.py command must specify the plugin function.
The plugin () function is performed against every line in the log data, while the pre () and post () functions are run when the script is started and stopped, respectively.
Each function can have any arguments defined for it, and these arguments can then be passed to the script using the plugin.arg option. Additionally, each function can have its own return values and actions defined for it.

Example 7.8. Simple Named Pipe Log Plug-in

def pre(myargs):
    retval = True
    myarg = myargs['argname']
    if isinstance(myarg, list): # handle list of values
    else: # handle single value
    if bad_problem:
        retval = False
    return retval

def plugin(line):
    retval = True
    # do something with line
    if something_is_bogus:
        retval = False
    return retval

def post(): # no arguments
    # do something
    # no return value

[1] Server performance suffers when log buffering is disabled on the access log, when the log level is changed on the error log, or with audit logging.
[2] The audit log is not enabled by default, so this log has to be enabled before a named pipe can be used to replace it.