15.2. Managing Logs

The Certificate System subsystem log files record events related to operations within that specific subsystem instance. For each subsystem, different logs are kept for issues such as installation, access, and web servers.
All subsystems have similar log configuration, options, and administrative paths.

15.2.1. An Overview of Log Settings

The way that logs are configured can affect Certificate System performance. For example, log file rotation keeps logs from becoming too large, which slows down subsystem performance. This section explains the different kinds of logs recorded by Certificate System subsystems and covers important concepts such as log file rotation, buffered logging, and available log levels.

15.2.1.1. Services That Are Logged

All major components and protocols of Certificate System log messages to log files. Table 15.1, “Services Logged” lists services that are logged by default. To view messages logged by a specific service, customize log settings accordingly. For details, see Section 15.3.1, “Viewing Logs in the Console”.

Table 15.1. Services Logged

Service Description
ACLs Logs events related to access control lists.
Administration Logs events related to administration activities, such as HTTPS communication between the Console and the instance.
All Logs events related to all the services.
Authentication Logs events related to activity with the authentication module.
Certificate Authority Logs events related to the Certificate Manager.
Database Logs events related to activity with the internal database.
HTTP
Logs events related to the HTTP activity of the server. Note that HTTP events are actually logged to the errors log belonging to the Apache server incorporated with the Certificate System to provide HTTP services.
Key Recovery Authority Logs events related to the KRA.
LDAP Logs events related to activity with the LDAP directory, which is used for publishing certificates and CRLs.
OCSP Logs events related to OCSP, such as OCSP status GET requests.
Others Logs events related to other activities, such as command-line utilities and other processes.
Request Queue Logs events related to the request queue activity.
User and Group Logs events related to users and groups of the instance.

15.2.1.2. Log Levels (Message Categories)

The different events logged by Certificate System services are determined by the log levels, which makes identifying and filtering events simpler. The different Certificate System log levels are listed in Table 15.2, “Log Levels and Corresponding Log Messages”.
Log levels are represented by numbers indicating how detailed the level of logging to be performed by the server should be.
A higher priority level means less detail because only events of high priority are logged.

Table 15.2. Log Levels and Corresponding Log Messages

Log level Message category Description
0-1 Tracing These messages contain finer-grained debugging information. This level should not be used regularly because it may impact the performance.
2-5 Debugging These messages contain debugging information. This level is not recommended for regular use because it generates too much information.
6-10 Informational These messages provide general information about the state of the Certificate System, including status messages such as Certificate System initialization complete and Request for operation succeeded.
11-15 Warning These messages are warnings only and do not indicate any failure in the normal operation of the server.
> 15 Failure These messages indicate errors and failures that prevent the server from operating normally, including failures to perform a certificate service operation (User authentication failed or Certificate revoked) and unexpected situations that can cause irrevocable errors (The server cannot send back the request it processed for a client through the same channel the request came from the client). Setting the level above 15 will minimize the logs, as only failures will be recorded.
Log levels can be used to filter log entries based on the severity of an event. The default log level is 10.
Log data can be extensive, especially at lower (more verbose) logging levels. Make sure that the host machine has sufficient disk space for all the log files. It is also important to define the logging level, log rotation, and server-backup policies appropriately so that all the log files are backed up and the host system does not get overloaded; otherwise, information can be lost.

15.2.1.3. Buffered and Unbuffered Logging

The Java subsystems support buffered logging for all types of logs. The server can be configured for either buffered or unbuffered logging.
If buffered logging is configured, the server creates buffers for the corresponding logs and holds the messages in the buffers for as long as possible. The server flushes out the messages to the log files only when one of the following conditions occurs:
  • The buffer gets full. The buffer is full when the buffer size is equal to or greater than the value specified by the bufferSize configuration parameter. The default value for this parameter is 512 KB.
  • The flush interval for the buffer is reached. The flush interval is reached when the time interval since the last buffer flush is equal to or greater than the value specified by the flushInterval configuration parameter. The default value for this parameter is 5 seconds.
  • When current logs are read from Console. The server retrieves the latest log when it is queried for current logs.
If the server is configured for unbuffered logging, the server flushes out messages as they are generated to the log files. Because the server performs an I/O operation (writing to the log file) each time a message is generated, configuring the server for unbuffered logging decreases performance.
Setting log parameters is described in Section 15.2.2, “Configuring Logs in the Console”.

15.2.1.4. Log File Rotation

The subsystem logs have an optional log setting that allows them to be rotated and start a new log file instead of letting log files grow indefinitely. Log files are rotated when either of the following occur:
  • The size limit for the corresponding file is reached. The size of the corresponding log file is equal to or greater than the value specified by the maxFileSize configuration parameter. The default value for this parameter is 100 KB.
  • The age limit for the corresponding file is reached. The corresponding log file is equal to or older than the interval specified by the rolloverInterval configuration parameter. The default value for this parameter is 2592000 seconds (every thirty days).

Note

Setting both these parameters to 0 effectively disables the log file rotation.
When a log file is rotated, the old file is named using the name of the file with an appended time stamp. The appended time stamp is an integer that indicates the date and time the corresponding active log file was rotated. The date and time have the forms YYYYMMDD (year, month, day) and HHMMSS (hour, minute, second).
Log files, especially the audit log file, contain critical information. These files should be periodically archived to some backup medium by copying the entire log directory to an archive medium.

Note

The Certificate System does not provide any tool or utility for archiving log files.
The Certificate System provides a command-line utility, signtool, that signs log files before archiving them as a means of tamper detection. For details, see Section 15.2.4.5, “Signing Log Files”.
Signing log files is an alternative to the signed audit logs feature. Signed audit logs create audit logs that are automatically signed with a subsystem signing certificate. See Section 15.2.4.3, “Configuring a Signed Audit Log in the Console” for details about signed audit logs.
Rotated log files are not deleted.

15.2.2. Configuring Logs in the Console

Logs can be configured through both the subsystem Console and through the subsystem's CS.cfg file. Specialized logs, such as signed audit logs and custom logs, can also be created through the Console or configuration file.
System, transaction, and audit logs can be configured through the subsystem Console for the CA, OCSP, TKS, and KRA subsystems. TPS logs are only configured through the configuration file.
  1. In the navigation tree of the Configuration tab, select Log.
  2. The Log Event Listener Management tab lists the currently configured listeners.
    To create a new log instance, click Add, and select a module plug-in from the list in the Select Log Event Listener Plug-in Implementation window.
  3. Set or modify the fields in the Log Event Listener Editor window. The different parameters are listed in Table 15.3, “Log Event Listener Fields”.

Table 15.3. Log Event Listener Fields

Field Description
Log Event Listener ID Gives the unique name that identifies the listener. The names can have any combination of letters (aA to zZ), digits (0 to 9), an underscore (_), and a hyphen (-), but it cannot contain other characters or spaces.
type Gives the type of log file. system creates error and system logs; transaction records audit logs.
enabled Sets whether the log is active. Only enabled logs actually record events. The value is either true or false.
level Sets the log level in the text field. The level must be manually entered in the field; there is no selection menu. The choices are Debug, Information, Warning, Failure, Misconfiguration, Catastrophe, and Security. For more information, see Section 15.2.1.2, “Log Levels (Message Categories)”.
fileName Gives the full path, including the file name, to the log file. The subsystem user should have read/write permission to the file.
bufferSize Sets the buffer size in kilobytes (KB) for the log. Once the buffer reaches this size, the contents of the buffer are flushed out and copied to the log file. The default size is 512 KB. For more information on buffered logging, see Section 15.2.1.3, “Buffered and Unbuffered Logging”.
flushInterval Sets the amount of time before the contents of the buffer are flushed out and added to the log file. The default interval is 5 seconds.
maxFileSize Sets the size, in kilobytes (KB), a log file can become before it is rotated. Once it reaches this size, the file is copied to a rotated file, and the log file is started new. For more information on log file rotation, see Section 15.2.1.4, “Log File Rotation”. The default size is 2000 KB.
rolloverInterval Sets the frequency for the server to rotate the active log file. The available options are hourly, daily, weekly, monthly, and yearly. The default is monthly. For more information, see Section 15.2.1.4, “Log File Rotation”.

15.2.3. Configuring Logs in the CS.cfg File

For instruction on how to configure logs by editing the CS.cfg file, see the Configuring Logs in the CS.cfg File section in the Red Hat Certificate System Planning, Installation, and Deployment Guide.

15.2.4. Managing Audit Logs

The audit log contains records for events that have been set up as recordable events. If the logSigning attribute is set to true, the audit log is signed with a log signing certificate belonging to the server. This certificate can be used by auditors to verify that the log has not been tampered with.
By default, regular audit logs are located in the /var/log/pki/instance_name/subsystem_name/ directory with other types of logs, while signed audit logs are written to /var/log/pki/instance_name/subsystem_name/signedAudit/. The default location for logs can be changed by modifying the configuration.
The signed audit log creates a log recording system events, and the events are selected from a list of potential events. When enabled, signed audit logs record a verbose set of messages about the selected event activity.
Signed audit logs are configured by default when the instance is first created, but it is possible to configure signed audits logs after installation. (See Section 15.2.4.2, “Enabling Signed Audit Logging after Installation”.) It is also possible to edit the configuration or change the signing certificates after configuration, as covered in Section 15.2.4.3, “Configuring a Signed Audit Log in the Console”.

15.2.4.1. A List of Audit Events

For a list of audit events in Certificate System, see Appendix E, Audit Events.

15.2.4.2. Enabling Signed Audit Logging after Installation

Signed audit logs can be enabled by default when an instance is first created by using the pki_audit_group deployment parameter with the pkispawn command. If, however, signed audit logs were not configured when an instance was created, they can be enabled afterwards by reassigning ownership of the audit log directory to the auditor system users group, such as pkiaudit.
  1. Stop the instance: 
    systemctl stop pki-tomcatd@instance_name.service
  2. Set the group ownership of the signed audit log directory to the PKI auditors operating system group, such as pkiaudit. This allows the users in the PKI auditors group to have the required read access to the signedAudit directory to verify the signatures on the log files. No user (except for the Certificate System user account, pkiuser) should have write access to the log files in this directory. 
    chgrp -R pkiaudit /var/log/pki/instance_name/subsystem_name/signedAudit
  3. Restart the instance: 
    systemctl start pki-tomcatd@instance_name.service

15.2.4.3. Configuring a Signed Audit Log in the Console

Signed audit logs are configured by default when the instance is first created, but it is possible to edit the configuration or change the signing certificates after configuration.

Note

Provide enough space in the file system for the signed audit logs, since they can be large.
A log is set to a signed audit log by setting the logSigning parameter to enable and providing the nickname of the certificate used to sign the log. A special log signing certificate is created when the subsystems are first configured.
Only a user with auditor privileges can access and view a signed audit log. Auditors can use the AuditVerify tool to verify that signed audit logs have not been tampered with.
The signed audit log is created and enabled when the subsystem is configured, but it needs additional configuration to begin creating and signing audit logs.
  1. Open the Console.

    Note

    To create or configure the audit log by editing the CS.cfg file, see the Configuring Logs in the CS.cfg File section in the Red Hat Certificate System Planning, Installation, and Deployment Guide.
  2. In the navigation tree of the Configuration tab, select Log.
  3. In the Log Event Listener Management tab, select the SignedAudit entry.
  4. Click Edit/View.
  5. There are three fields which must be reset in the Log Event Listener Editor window.
    • Fill in the signedAuditCertNickname. This is the nickname of the certificate used to sign audit logs. An audit signing certificate is created when the subsystem is configured; it has a nickname like auditSigningCert cert-instance_name subsystem_name.

      Note

      To get the audit signing certificate nickname, list the certificates in the subsystem's certificate database using certutil. For example: 
      certutil -L -d /var/lib/pki-tomcat/alias

Certificate Authority - Example Domain    CT,c,
subsystemCert cert-pki-tomcat             u,u,u
Server-Cert cert-pki-tomcat               u,u,u
auditSigningCert cert-pki-tomcat CA       u,u,Pu
    • Set the logSigning field to true to enable signed logging.
    • Set any events which are logged to the audit log. Appendix E, Audit Events lists the loggable events. Log events are separated by commas with no spaces.
  6. Set any other settings for the log, such as the file name, the log level, the file size, or the rotation schedule.

    Note

    By default, regular audit logs are located in the /var/log/pki/instance_name/subsystem_name/ directory with other types of logs, while signed audit logs are written to /var/log/pki/instance_name/subsystem_name/signedAudit/. The default location for logs can be changed by modifying the configuration.
  7. Save the log configuration.
After enabling signed audit logging, assign auditor users by creating the user and assigning that entry to the auditor group. Members of the auditor group are the only users who can view and verify the signed audit log. See Section 14.3.2.1, “Creating Users” for details about setting up auditors.
Auditors can verify logs by using the AuditVerify tool. See the AuditVerify(1) man page for details about using this tool.

15.2.4.4. Handling Audit Logging Failures

There are events that could cause the audit logging function to fail, so events cannot be written to the log. For example, audit logging can fail when the file system containing the audit log file is full or when the file permissions for the log file are accidentally changed. If audit logging fails, the Certificate System instance shuts down in the following manner.
  • Servlets are disabled and will not process new requests.
  • All pending and new requests are killed.
  • The subsystem is shut down.
When this happens, administrators and auditors should work together with the operating system administrator to resolve the disk space or file permission issues. When the IT problem is resolved, the auditor should make sure that the last audit log entries are signed. If not, they should be preserved by manual signing (Section 15.2.4.5, “Signing Log Files”), archived, and removed to prevent audit verification failures in the future. When this is completed, the administrators can restart the Certificate System.

15.2.4.5. Signing Log Files

The Certificate System can digitally sign log files before they are archived or distributed for audit purposes. This feature allows files to be checked for tampering.
This is an alternative to the signed audit logs feature. The signed audit log feature creates audit logs that are automatically signed; this tool manually signs archived logs. See Section 15.2.4.3, “Configuring a Signed Audit Log in the Console” for details about signed audit logs.
For signing log files, use a command-line utility called the Signing Tool (signtool). For details about this utility, see http://www.mozilla.org/projects/security/pki/nss/tools/.
The utility uses information in the certificate, key, and security module databases of the subsystem instance.
As a user with auditor privilegesuse the signtool command to sign the log directories: 
signtool -d secdb_dir -k cert_nickname -Z output input
  • secdb_dir specifies the path to the directory that contains the certificate, key, and security module databases for the CA.
  • cert_nickname specifies the nickname of the certificate to use for signing.
  • output specifies the name of the JAR file (a signed zip file).
  • input specifies the path to the directory that contains the log files.

15.2.4.6. Filtering Audit Events

In Certificate System administrators can set filters to configure which audit events will be logged in the audit file based on the event attributes.
The format of the filters is the same as for LDAP filters. However, Certificate System only supports the following filters:

Table 15.4. Supported Audit Event Filters

Type Format Example
Presence (attribute=*) (ReqID=*)
Equality (attribute=value) (Outcome=Failure)
Substring (attribute=initial*any*...*any*final) (SubjectID=*admin*)
AND operation (&(filter_1)(filter_2)...(filter_n)) (&(SubjectID=admin)(Outcome=Failure))
OR operation (|(filter_1)(filter_2)...(filter_n)) (|(SubjectID=admin)(Outcome=Failure))
NOT operation (!(filter)) (!(SubjectID=admin))
For further details on LDAP filters, see the Using Compound Search Filters in the Red Hat Directory Server Administration Guide.

Example 15.5. Filtering Audit Events

To log only failed events for profile certificate requests and events for processed certificates requests that have the InfoName field set to rejectReadon or cancelReason:
  1. Edit the /var/lib/pki/instance_name/subsystem_type/conf/CS.cfg file and set the following parameters: 
    log.instance.SignedAudit.filters.PROFILE_CERT_REQUEST=(Outcome=Failure)
log.instance.SignedAudit.filters.CERT_REQUEST_PROCESSED=(|(InfoName=rejectReason)(InfoName=cancelReason))
  2. Restart Certificate System: 
    # systemctl restart pki-tomcatd@instance_name.service

15.2.5. Managing Log Modules

The types of logs that are allowed and their behaviors are configured through log module plug-ins. New logging modules can be created and used to make custom logs.
New log plug-in modules can be registered through the Console. Registering a new module involves specifying the name of the module and the full name of the Java™ class that implements the log interface.
Before registering a plug-in module, put the Java™ class for the module in the classes directory; the implementation must be on the class path.
To register a log plug-in module with a subsystem instance:
  1. Create the custom job class. For this example, the custom log plug-in is called MyLog.java.
  2. Compile the new class into the lib directory of the instance. 
    javac -d . /var/lib/pki/pki-tomcat/lib -classpath $CLASSPATH MyLog.java
  3. Create a directory in the CA's WEB-INF web directory to hold the custom classes, so that the CA can access them. 
    mkdir /var/lib/pki/pki-tomcat/webapps/ca/WEB-INF/classes
  4. Set the owner to the Certificate System system user (pkiuser). 
    chown -R pkiuser:pkiuser /var/lib/pki/pki-tomcat/lib
  5. Register the plug-in.
    1. Log into the Console.
    2. In the Configuration tab, select Logs from the navigation tree. Then select the Log Event Listener Plug-in Registration tab.
    3. Click Register.
      The Register Log Event Listener Plug-in Implementation window appears.
    4. Give the name for the plug-in module and the Java™ class name.
      The Java™ class name is the full path to the implementing Java™ class. If this class is part of a package, include the package name. For example, registering a class named customLog in a package named com.customplugins, the class name would be com.customplugins.customLog.
    5. Click OK.
Unwanted log plug-in modules can be deleted through the Console. Before deleting a module, delete all the listeners based on this module; see Section 15.2.1.4, “Log File Rotation”.