Chapter 14. Configuring Data Grid Server logging

Data Grid Server uses Apache Log4j 2 to provide configurable logging mechanisms that capture details about the environment and record cache operations for troubleshooting purposes and root cause analysis.

14.1. Data Grid Server log files

Data Grid writes server logs to the following files in the $RHDG_HOME/server/log directory:

server.log
Messages in human readable format, including boot logs that relate to the server startup.
Data Grid creates this file when you start the server.
server.log.json
Messages in JSON format that let you parse and analyze Data Grid logs.
Data Grid creates this file when you enable the JSON-FILE appender.

14.1.1. Configuring Data Grid Server logs

Data Grid uses Apache Log4j technology to write server log messages. You can configure server logs in the log4j2.xml file.

Procedure

  1. Open $RHDG_HOME/server/conf/log4j2.xml with any text editor.
  2. Change server logging as appropriate.
  3. Save and close log4j2.xml.

Additional resources

14.1.2. Log levels

Log levels indicate the nature and severity of messages.

Log levelDescription

TRACE

Fine-grained debug messages, capturing the flow of individual requests through the application.

DEBUG

Messages for general debugging, not related to an individual request.

INFO

Messages about the overall progress of applications, including lifecycle events.

WARN

Events that can lead to error or degrade performance.

ERROR

Error conditions that might prevent operations or activities from being successful but do not prevent applications from running.

FATAL

Events that could cause critical service failure and application shutdown.

In addition to the levels of individual messages presented above, the configuration allows two more values: ALL to include all messages, and OFF to exclude all messages.

14.1.3. Data Grid logging categories

Data Grid provides categories for INFO, WARN, ERROR, FATAL level messages that organize logs by functional area.

org.infinispan.CLUSTER
Messages specific to Data Grid clustering that include state transfer operations, rebalancing events, partitioning, and so on.
org.infinispan.CONFIG
Messages specific to Data Grid configuration.
org.infinispan.CONTAINER
Messages specific to the data container that include expiration and eviction operations, cache listener notifications, transactions, and so on.
org.infinispan.PERSISTENCE
Messages specific to cache loaders and stores.
org.infinispan.SECURITY
Messages specific to Data Grid security.
org.infinispan.SERVER
Messages specific to Data Grid servers.
org.infinispan.XSITE
Messages specific to cross-site replication operations.

14.1.4. Log appenders

Log appenders define how Data Grid Server records log messages.

CONSOLE
Write log messages to the host standard out (stdout) or standard error (stderr) stream.
Uses the org.apache.logging.log4j.core.appender.ConsoleAppender class by default.
FILE
Write log messages to a file.
Uses the org.apache.logging.log4j.core.appender.RollingFileAppender class by default.
JSON-FILE
Write log messages to a file in JSON format.
Uses the org.apache.logging.log4j.core.appender.RollingFileAppender class by default.

14.1.5. Log pattern formatters

The CONSOLE and FILE appenders use a PatternLayout to format the log messages according to a pattern.

An example is the default pattern in the FILE appender:
%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p (%t) [%c{1}] %m%throwable%n

  • %d{yyyy-MM-dd HH:mm:ss,SSS} adds the current time and date.
  • %-5p specifies the log level, aligned to the right.
  • %t adds the name of the current thread.
  • %c{1} adds the short name of the logging category.
  • %m adds the log message.
  • %throwable adds the exception stack trace.
  • %n adds a new line.

Patterns are fully described in the PatternLayout documentation .

14.1.6. Enabling the JSON log handler

Data Grid Server provides a log handler to write messages in JSON format.

Prerequisites

  • Stop Data Grid Server if it is running.
    You cannot dynamically enable log handlers.

Procedure

  1. Open $RHDG_HOME/server/conf/log4j2.xml with any text editor.

  2. Uncomment the JSON-FILE appender and comment out the FILE appender: 

    <!--<AppenderRef ref="FILE"/>-->
<AppenderRef ref="JSON-FILE"/>
  3. Optionally configure the JSON appender and JSON layout as required.
  4. Save and close log4j2.xml.

When you start Data Grid, it writes each log message as a JSON map in the following file:
$RHDG_HOME/server/log/server.log.json

Additional resources

14.2. Access logs

Access logs record all inbound client requests for Hot Rod and REST endpoints to files in the $RHDG_HOME/server/log directory.

org.infinispan.HOTROD_ACCESS_LOG
Logging category that writes Hot Rod access messages to a hotrod-access.log file.
org.infinispan.REST_ACCESS_LOG
Logging category that writes REST access messages to a rest-access.log file.

14.2.1. Enabling access logs

To record Hot Rod and REST endpoint access messages, you need to enable the logging categories in log4j2.xml.

Procedure

  1. Open $RHDG_HOME/server/conf/log4j2.xml with any text editor.
  2. Change the level for the org.infinispan.HOTROD_ACCESS_LOG and org.infinispan.REST_ACCESS_LOG logging categories to TRACE.
  3. Save and close log4j2.xml. 
<Logger name="org.infinispan.HOTROD_ACCESS_LOG" additivity="false" level="TRACE">
   <AppenderRef ref="HR-ACCESS-FILE"/>
</Logger>

14.2.2. Access log properties

The default format for access logs is as follows: 

%X{address} %X{user} [%d{dd/MMM/yyyy:HH:mm:ss Z}] &quot;%X{method} %m
%X{protocol}&quot; %X{status} %X{requestSize} %X{responseSize} %X{duration}%n

The preceding format creates log entries such as the following:

127.0.0.1 - [DD/MM/YYYY:HH:MM:SS +0000] "PUT /rest/v2/caches/default/key HTTP/1.1" 404 5 77 10

Logging properties use the %X{name} notation and let you modify the format of access logs. The following are the default logging properties:

PropertyDescription

address

Either the X-Forwarded-For header or the client IP address.

user

Principal name, if using authentication.

method

Method used. PUT, GET, and so on.

protocol

Protocol used. HTTP/1.1, HTTP/2, HOTROD/2.9, and so on.

status

An HTTP status code for the REST endpoint. OK or an exception for the Hot Rod endpoint.

requestSize

Size, in bytes, of the request.

responseSize

Size, in bytes, of the response.

duration

Number of milliseconds that the server took to handle the request.

Tip

Use the header name prefixed with h: to log headers that were included in requests; for example, %X{h:User-Agent}.

14.3. Audit logs

Audit logs let you track changes to your Data Grid Server deployment so you know when changes occur and which users make them. Enable and configure audit logging to record server configuration events and administrative operations.

org.infinispan.AUDIT
Logging category that writes security audit messages to an audit.log file in the $RHDG_HOME/server/log directory.

14.3.1. Enabling audit logging

To record security audit messages, you need to enable the logging category in log4j2.xml.

Procedure

  1. Open $RHDG_HOME/server/conf/log4j2.xml with any text editor.
  2. Change the level for the org.infinispan.AUDIT logging category to INFO.
  3. Save and close log4j2.xml. 
<!-- Set to INFO to enable audit logging -->
<Logger name="org.infinispan.AUDIT" additivity="false" level="INFO">
   <AppenderRef ref="AUDIT-FILE"/>
</Logger>

14.3.2. Configuring audit logging appenders

Apache Log4j provides different appenders that you can use to send audit messages to a destination other than the default log file. For instance, if you want to send audit logs to a syslog daemon, JDBC database, or Apache Kafka server, you can configure an appender in log4j2.xml.

Procedure

  1. Open $RHDG_HOME/server/conf/log4j2.xml with any text editor.

  2. Comment or remove the default AUDIT-FILE rolling file appender. 

    <!--RollingFile name="AUDIT-FILE"
  ...
</RollingFile-->

  3. Add the desired logging appender for audit messages.

    For example, you could add a logging appender for a Kafka server as follows: 

    <Kafka name="AUDIT-KAFKA" topic="audit">
  <PatternLayout pattern="%date %message"/>
  <Property name="bootstrap.servers">localhost:9092</Property>
</Kafka>
  4. Save and close log4j2.xml.

Additional resources

14.3.3. Using custom audit logging implementations

You can create custom implementations of the org.infinispan.security.AuditLogger API if configuring Log4j appenders does not meet your needs.

Prerequisites

  • Implement org.infinispan.security.AuditLogger as required and package it in a JAR file.

Procedure

  1. Add your JAR to the server/lib directory in your Data Grid Server installation.

  2. Specify the fully qualified class name of your custom audit logger as the value for the audit-logger attribute on the authorization element in your cache container security configuration.

    For example, the following configuration defines my.package.CustomAuditLogger as the class for logging audit messages: 

    <infinispan>
   <cache-container>
      <security>
         <authorization audit-logger="my.package.CustomAuditLogger"/>
      </security>
   </cache-container>
</infinispan>

Additional resources