Chapter 15. Configuring Subsystem Logs

The Certificate System subsystems create log files that record events related to activities, such as administration, communications using any of the protocols the server supports, and various other processes employed by the subsystems. While a subsystem instance is running, it keeps a log of information and error messages on all the components it manages. Additionally, the Apache and Tomcat web servers generate error and access logs.
Each subsystem instance maintains its own log files for installation, audit, and other logged functions.
Log plug-in modules are listeners which are implemented as Java™ classes and are registered in the configuration framework.
All the log files and rotated log files, except for audit logs, are located in whatever directory was specified in pki_subsystem_log_path when the instance was created with pkispawn. Regular audit logs are located in the log directory with other types of logs, while signed audit logs are written to /var/log/pki/instance_name/subsystem_name/signedAudit. The default location for logs can be changed by modifying the configuration.

15.1. About Certificate System Logs

Certificate System subsystems keep several different kinds of logs, which provide specific information depending on the type of subsystem, types of services, and individual log settings. The kinds of logs that can be kept for an instance depend on the kind of subsystem that it is.

15.1.1. System Log

Subsystem logs are kept for the CA, OCSP, KRA, and TKS subsystems.
This log, system, records information about requests to the server (all HTTP and HTTPS requests) and the responses from the server. Information recorded in this log includes the IP address (both IPv4 and IPv6) of the client machine that accessed the server; operations performed, such as search, add, and edit; and the result of the access, such as the number of entries returned: 
id_number processor - [date:time] [number_of_operations] [result] servlet: message

Example 15.1. TKS System Log

10439.http-13443-Processor25 - [19/May/2020:14:16:51 CDT] [11] [3] UGSubsystem: Get User Error User not found
This log is on by default.

15.1.2. Transactions Log

Transaction logs are kept for the CA, OCSP, KRA, and TKS subsystems.
This log, transactions, records any kind of operation performed or submitted to the subsystem. 
id_number.processor - [date:time] [number_of_operations] [result] servlet: message
These messages are specific to the certificate service, such as the CA receiving certificate requests, the KRA archiving or retrieving keys, and the TKS registering a new TPS. This log can also be used to detect any unauthorized access or activity.

Example 15.2. Transactions Log

11438.http-8443-Processor25 - [27/May/2020:07:56:18 CDT] [1] [1] archival reqID 4 fromAgent agentID: CA-server.example.com-8443 authenticated by noAuthManager is completed DN requested: UID=recoverykey,E=recoverykey@email.com,CN=recover key serial number: 0x3
This log is on by default.

15.1.3. Debug Logs

Debug logs, which are enabled by default, are maintained for all subsystems, with varying degrees and types of information.
Debug logs for each subsystem record much more detailed information than system, transaction, and access logs. Debug logs contain very specific information for every operation performed by the subsystem, including plug-ins and servlets which are run, connection information, and server request and response messages.
The general types of services which are recorded to the debug log are briefly discussed in Section 15.2.1.1, “Services That Are Logged”. These services include authorization requests, processing certificate requests, certificate status checks, and archiving and recovering keys, and access to web services.
The debug logs for the CA, OCSP, KRA, and TKS record detailed information about the processes for the subsystem. Each log entry has the following format: 
[date:time] [processor]: servlet: message
The message can be a return message from the subsystem or contain values submitted to the subsystem.
For example, the TKS records this message for connecting to an LDAP server: 
[10/Jun/2020:05:14:51][main]: Established LDAP connection using basic authentication to host localhost port 389 as cn=Directory Manager
The processor is main, and the message is the message from the server about the LDAP connection, and there is no servlet.
The CA, on the other hand, records information about certificate operations as well as subsystem connections: 
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.requestowner$ value=KRA-server.example.com-8443
In this case, the processor is the HTTP protocol over the CA's agent port, while it specifies the servlet for handling profiles and contains a message giving a profile parameter (the subsystem owner of a request) and its value (that the KRA initiated the request).

Example 15.3. CA Certificate Request Log Messages

[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.profileapprovedby$ value=admin
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.cert_request$ value=MIIBozCCAZ8wggEFAgQqTfoHMIHHgAECpQ4wDDEKMAgGA1UEAxMBeKaBnzANBgkqhkiG9w0BAQEFAAOB...
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.profile$ value=true
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.cert_request_type$ value=crmf
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.requestversion$ value=1.0.0
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.req_locale$ value=en
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.requestowner$ value=KRA-server.example.com-8443
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.dbstatus$ value=NOT_UPDATED
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.subject$ value=uid=jsmith, e=jsmith@example.com
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.requeststatus$ value=begin
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.auth_token.user$ value=uid=KRA-server.example.com-8443,ou=People,dc=example,dc=com
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.req_key$ value=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDreuEsBWq9WuZ2MaBwtNYxvkLP^M
HcN0cusY7gxLzB+XwQ/VsWEoObGldg6WwJPOcBdvLiKKfC605wFdynbEgKs0fChV^M
k9HYDhmJ8hX6+PaquiHJSVNhsv5tOshZkCfMBbyxwrKd8yZ5G5I+2gE9PUznxJaM^M
HTmlOqm4HwFxzy0RRQIDAQAB
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.auth_token.authmgrinstname$ value=raCertAuth
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.auth_token.uid$ value=KRA-server.example.com-8443
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.auth_token.userid$ value=KRA-server.example.com-8443
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.requestor_name$ value=
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.profileid$ value=caUserCert
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.auth_token.userdn$ value=uid=KRA-server.example.com-4747,ou=People,dc=example,dc=com
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.requestid$ value=20
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.auth_token.authtime$ value=1212782378071
[06/Jun/2020:14:59:38][http-8443;-Processor24]: ProfileSubmitServlet: key=$request.req_x509info$ value=MIICIKADAgECAgEAMA0GCSqGSIb3DQEBBQUAMEAxHjAcBgNVBAoTFVJlZGJ1ZGNv^M
bXB1dGVyIERvbWFpbjEeMBwGA1UEAxMVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB4X^M
DTA4MDYwNjE5NTkzOFoXDTA4MTIwMzE5NTkzOFowOzEhMB8GCSqGSIb3DQEJARYS^M
anNtaXRoQGV4YW1wbGUuY29tMRYwFAYKCZImiZPyLGQBARMGanNtaXRoMIGfMA0G^M
CSqGSIb3DQEBAQUAA4GNADCBiQKBgQDreuEsBWq9WuZ2MaBwtNYxvkLPHcN0cusY^M
7gxLzB+XwQ/VsWEoObGldg6WwJPOcBdvLiKKfC605wFdynbEgKs0fChVk9HYDhmJ^M
8hX6+PaquiHJSVNhsv5tOshZkCfMBbyxwrKd8yZ5G5I+2gE9PUznxJaMHTmlOqm4^M
HwFxzy0RRQIDAQABo4HFMIHCMB8GA1UdIwQYMBaAFG8gWeOJIMt+aO8VuQTMzPBU^M
78k8MEoGCCsGAQUFBwEBBD4wPDA6BggrBgEFBQcwAYYuaHR0cDovL3Rlc3Q0LnJl^M
ZGJ1ZGNvbXB1dGVyLmxvY2FsOjkwODAvY2Evb2NzcDAOBgNVHQ8BAf8EBAMCBeAw^M
HQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMCQGA1UdEQQdMBuBGSRyZXF1^M
ZXN0LnJlcXVlc3Rvcl9lbWFpbCQ=
Likewise, the OCSP shows OCSP request information: 
[07/Jul/2020:06:25:40][http-11180-Processor25]: OCSPServlet: OCSP Request:
[07/Jul/2020:06:25:40][http-11180-Processor25]: OCSPServlet:
MEUwQwIBADA+MDwwOjAJBgUrDgMCGgUABBSEWjCarLE6/BiSiENSsV9kHjqB3QQU

15.1.3.1. Installation Logs

All subsystems keep an install log.
Every time a subsystem is created either through the initial installation or creating additional instances with pkispawn, an installation file with the complete debug output from the installation, including any errors and, if the installation is successful, the URL and PIN to the configuration interface for the instance. The file is created in the /var/log/pki/ directory for the instance with a name in the form pki-subsystem_name-spawn.timestamp.log.
Each line in the install log follows a step in the installation process.

Example 15.4. CA Install Log

...
2015-07-22 20:43:13 pkispawn    : INFO     ... finalizing 'pki.server.deployment.scriptlets.finalization'
2015-07-22 20:43:13 pkispawn    : INFO     ....... cp -p /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg /var/log/pki/pki-tomcat/ca/archive/spawn_deployment.cfg.20150722204136
2015-07-22 20:43:13 pkispawn    : DEBUG    ........... chmod 660 /var/log/pki/pki-tomcat/ca/archive/spawn_deployment.cfg.20150722204136
2015-07-22 20:43:13 pkispawn    : DEBUG    ........... chown 26445:26445 /var/log/pki/pki-tomcat/ca/archive/spawn_deployment.cfg.20150722204136
2015-07-22 20:43:13 pkispawn    : INFO     ....... generating manifest file called '/etc/sysconfig/pki/tomcat/pki-tomcat/ca/manifest'
2015-07-22 20:43:13 pkispawn    : INFO     ....... cp -p /etc/sysconfig/pki/tomcat/pki-tomcat/ca/manifest /var/log/pki/pki-tomcat/ca/archive/spawn_manifest.20150722204136
2015-07-22 20:43:13 pkispawn    : DEBUG    ........... chmod 660 /var/log/pki/pki-tomcat/ca/archive/spawn_manifest.20150722204136
2015-07-22 20:43:13 pkispawn    : DEBUG    ........... chown 26445:26445 /var/log/pki/pki-tomcat/ca/archive/spawn_manifest.20150722204136
2015-07-22 20:43:13 pkispawn    : INFO     ....... executing 'systemctl enable pki-tomcatd.target'
2015-07-22 20:43:13 pkispawn    : INFO     ....... executing 'systemctl daemon-reload'
2015-07-22 20:43:13 pkispawn    : INFO     ....... executing 'systemctl restart pki-tomcatd@pki-tomcat.service'
2015-07-22 20:43:14 pkispawn    : INFO     END spawning subsystem 'CA' of instance 'pki-tomcat'
2015-07-22 20:43:14 pkispawn    : DEBUG

15.1.3.2. Tomcat Error and Access Logs

The CA, KRA, OCSP, TKS, and TPS subsystems use a Tomcat web server instance for their agent and end-entities' interfaces.
Error and access logs are created by the Tomcat web server, which are installed with the Certificate System and provide HTTP services. The error log contains the HTTP error messages the server has encountered. The access log lists access activity through the HTTP interface.
Logs created by Tomcat:
  • admin.timestamp
  • catalina.timestamp
  • catalina.out
  • host-manager.timestamp
  • localhost.timestamp
  • localhost_access_log.timestamp
  • manager.timestamp
These logs are not available or configurable within the Certificate System; they are only configurable within Apache or Tomcat. See the Apache documentation for information about configuring these logs.

15.1.3.3. Self-Tests Log

The self-tests log records information obtained during the self-tests run when the server starts or when the self-tests are manually run. The tests can be viewed by opening this log. This log is not configurable through the Console, it can only be configured by changing settings in the CS.cfg file. For instruction on how to configure logs by editing the CS.cfg file, see the Enabling the Publishing Queue section in the Red Hat Certificate System Planning, Installation, and Deployment Guide.
The information about logs in this section does not pertain to this log. See Section 13.9, “Running Self-Tests” for more information about self-tests.