Chapter 29. Security for Cluster Traffic

29.1. Node Authentication and Authorization (Remote Client-Server Mode)

29.1.1. Node Authentication and Authorization (Remote Client-Server Mode)

Security can be enabled at node level via SASL protocol, which enables node authentication against a security realm. This requires nodes to authenticate each other when joining or merging with a cluster. For detailed information about security realms, see About Security Realms.

The following example depicts the <sasl /> element, which leverages the SASL protocol. Both DIGEST-MD5 or GSSAPI mechanisms are currently supported.

Configure SASL Authentication

        <!-- Additional configuration information here -->
        <security-realm name="ClusterRealm">
                <properties path="" relative-to="jboss.server.config.dir"/>
                    <properties path="" relative-to="jboss.server.config.dir"/>
        <!-- Additional configuration information here -->

<stack name="udp">
    <!-- Additional configuration information here -->
    <sasl mech="DIGEST-MD5" security-realm="ClusterRealm" cluster-role="cluster">
        <property name="client_name">node1</property>
        <property name="client_password">password</property>
    <!-- Additional configuration information here -->

In the provided example, the nodes use the DIGEST-MD5 mechanism to authenticate against the ClusterRealm. In order to join, nodes must have the cluster role.

The cluster-role attribute determines the role all nodes must belong to in the security realm in order to JOIN or MERGE with the cluster. Unless it has been specified, the cluster-role attribute is the name of the clustered <cache-container> by default. Each node identifies itself using the client-name property. If none is specified, the hostname on which the server is running will be used.

This name can also be overridden by specifying the system property that can be overridden on the command line. For example:


JGroups AUTH protocol is not integrated with security realms, and its use is not advocated for Red Hat JBoss Data Grid.

29.1.2. Configure Node Authentication for Cluster Security (DIGEST-MD5)

The following example demonstrates how to use DIGEST-MD5 with a properties-based security realm, with a dedicated realm for cluster node.

Using the DIGEST-MD5 Mechanism

             <security-realm name="ClusterRealm">
                     <properties path="" relative-to="jboss.server.config.dir"/>
                     <properties path="" relative-to="jboss.server.config.dir"/>
<subsystem xmlns="urn:infinispan:server:jgroups:8.0" default-stack="${jboss.default.jgroups.stack:udp}">
     <stack name="udp">
         <transport type="UDP" socket-binding="jgroups-udp"/>
         <protocol type="PING"/>
         <protocol type="MERGE2"/>
         <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
         <protocol type="FD_ALL"/>
         <protocol type="pbcast.NAKACK"/>
         <protocol type="UNICAST2"/>
         <protocol type="pbcast.STABLE"/>
         <protocol type="pbcast.GMS"/>
         <protocol type="UFC"/>
         <protocol type="MFC"/>
         <protocol type="FRAG3"/>
         <protocol type="RSVP"/>
         <sasl security-realm="ClusterRealm" mech="DIGEST-MD5">
             <property name="client_password>...</property>
<subsystem xmlns="urn:infinispan:server:core:8.4" default-cache-container="clustered">
     <cache-container name="clustered" default-cache="default">
         <transport executor="infinispan-transport" lock-timeout="60000" stack="udp"/>
         <!-- various clustered cache definitions here -->

In the provided example, supposing the hostnames of the various nodes are node001, node002, node003, the will contain:

  • node001=/<node001passwordhash>/
  • node002=/<node002passwordhash>/
  • node003=/<node003passwordhash>/

The will contain:

  • node001=clustered
  • node002=clustered
  • node003=clustered

To generate these values, the following script can be used:

$ -up -gp -r ClusterRealm -u node001 -g clustered -p <password>

The MD5 password hash of the node must also be placed in the "client_password" property of the <sasl/> element.

<property name="client_password>...</property>

To increase security, it is recommended that this password be stored using a Vault. For more information about vault expressions, see the Red Hat Enterprise Application Platform Security Guide

Once node security has been set up as discussed here, the cluster coordinator will validate each JOINing and MERGEing node’s credentials against the realm before letting the node become part of the cluster view.

29.1.3. Configure Node Authentication for Cluster Security (GSSAPI/Kerberos)

When using the GSSAPI mechanism, the client_name is used as the name of a Kerberos-enabled login module defined within the security domain subsystem. For a full procedure on how to do this, see Configure Hot Rod Authentication (GSSAPI/Kerberos).

Using the Kerberos Login Module

<security-domain name="krb-node0" cache-type="default">
        <login-module code="Kerberos" flag="required">
            <module-option name="storeKey" value="true"/>
            <module-option name="useKeyTab" value="true"/>
            <module-option name="refreshKrb5Config" value="true"/>
            <module-option name="principal" value="jgroups/node0/clustered@INFINISPAN.ORG"/>
            <module-option name="keyTab" value="${jboss.server.config.dir}/keytabs/jgroups_node0_clustered.keytab"/>
            <module-option name="doNotPrompt" value="true"/>

The following property must be set in the <sasl/> element to reference it:

<sasl <!-- Additional configuration information here --> >
     <property name="login_module_name">
     		<!-- Additional configuration information here -->

As a result, the authentication section of the security realm is ignored, as the nodes will be validated against the Kerberos Domain Controller. The authorization configuration is still required, as the node principal must belong to the required cluster-role.

In all cases, it is recommended that a shared authorization database, such as LDAP, be used to validate node membership in order to simplify administration.

By default, the principal of the joining node must be in the following format:


29.2. Configure Node Security in Library Mode

29.2.1. Configure Node Security in Library Mode

In Library mode, node authentication is configured directly in the JGroups configuration. JGroups can be configured so that nodes must authenticate each other when joining or merging with a cluster. The authentication uses SASL and is enabled by adding the SASL protocol to your JGroups XML configuration.

SASL relies on JAAS notions, such as CallbackHandlers, to obtain certain information necessary for the authentication handshake. Users must supply their own CallbackHandlers on both client and server sides.


The JAAS API is only available when configuring user authentication and authorization, and is not available for node security.


In the provided example, CallbackHandler classes are examples only, and not contained in the Red Hat JBoss Data Grid release. Users must provide the appropriate CallbackHandler classes for their specific LDAP implementation.

Setting Up SASL Authentication in JGroups

    sasl_props="" />

The above example uses the DIGEST-MD5 mechanism. Each node must declare the user and password it will use when joining the cluster.


The SASL protocol must be placed before the GMS protocol in order for authentication to take effect.

29.2.2. Simple Authorizing Callback Handler

For instances where a more complex Kerberos or LDAP approach is not needed the SimpleAuthorizingCallbackHandler class may be used. To enable this set both the server_callback_handler and the client_callback_handler to org.jgroups.auth.sasl.SimpleAuthorizingCallbackHandler , as seen in the below example:

  sasl_props="" />

The SimpleAuthorizingCallbackHandler may be configured either programmatically, by passing the constructor an instance of of java.util.Properties , or via standard Java system properties, set on the command line using the -DpropertyName=propertyValue notation. The following properties are available:

  • - the path to a property file which contains principal/credential mappings represented as principal=password .
  • sasl.local.principal - the name of the principal that is used to identify the local node. It must exist in the file.
  • - (optional) the path to a property file which contains principal/roles mappings represented as principal=role1,role2,role3 .
  • sasl.role - (optional) if present, authorizes joining nodes only if their principal is.
  • sasl.realm - (optional) the name of the realm to use for the SASL mechanisms that require it

29.2.3. Configure Node Authentication for Library Mode (DIGEST-MD5)

The behavior of a node differs depending on whether it is the coordinator node or any other node. The coordinator acts as the SASL server, with the joining or merging nodes behaving as SASL clients. When using the DIGEST-MD5 mechanism in Library mode, the server and client callback must be specified so that the server and client are aware of how to obtain the credentials. Therefore, two CallbackHandlers are required:

  • The server_callback_handler_class is used by the coordinator.
  • The client_callback_handler_class is used by other nodes.

The following example demonstrates these CallbackHandlers.

Callback Handlers


JGroups is designed so that all nodes are able to act as coordinator or client depending on cluster behavior, so if the current coordinator node goes down, the next node in the succession chain will become the coordinator. Given this behavior, both server and client callback handlers must be identified within SASL for Red Hat JBoss Data Grid implementations.

29.2.4. Configure Node Authentication for Library Mode (GSSAPI)

When performing node authentication in Library mode using the GSSAPI mechanism, the login_module_name parameter must be specified instead of callback.

This login module is used to obtain a valid Kerberos ticket, which is used to authenticate a client to the server. The server_name must also be specified, as the client principal is constructed as jgroups/$server_name@REALM .

Specifying the login module and server on the coordinator node

         server_callback_handler_class="" />

On the coordinator node, the server_callback_handler_class must be specified for node authorization. This will determine if the authenticated joining node has permission to join the cluster.


The server principal is always constructed as jgroups/server_name, therefore the server principal in Kerberos must also be jgroups/server_name. For example, if the server name in Kerberos is jgroups/node1/mycache, then the server name must be node1/mycache.

29.3. JGroups Encryption

29.3.1. JGroups Encryption

JGroups includes the SYM_ENCRYPT and ASYM_ENCRYPT protocols to provide encryption for cluster traffic.


The ENCRYPT protocol has been deprecated and should not be used in production environments. It is recommended to use either SYM_ENCRYPT or ASYM_ENCRYPT

By default, both of these protocols only encrypt the message body; they do not encrypt message headers. To encrypt the entire message, including all headers, as well as destination and source addresses, the property encrypt_entire_message must be true. When defining these protocols they should be placed directly under NAKACK2.

Both protocols may be used to encrypt and decrypt communication in JGroups, and are used in the following ways:

  • SYM_ENCRYPT: Configured with a secret key in a keystore using the JCEKS store type.
  • ASYM_ENCRYPT: Configured with algorithms and key sizes. In this scenario the secret key is not retrieved from the keystore, but instead generated by the coordinator and distributed to new members. Once a member joins the cluster they send a request for the secret key to the coordinator; the coordinator responds with the secret key back to the new member encrypted with the member’s public key.

Each message is identified as encrypted with a specific encryption header identifying the encrypt header and an MD5 digest identifying the version of the key being used to encrypt and decrypt messages.

29.3.2. Configuring JGroups Encryption Protocols

JGroups encryption protocols are placed in the JGroups configuration file, and there are three methods of including this file depending on how JBoss Data Grid is in use:

  • Standard Java properties can also be used in the configuration, and it is possible to pass the path to JGroups configuration via the -D option during start up.
  • The default, pre-configured JGroups files are packaged in infinispan-embedded.jar , alternatively, you can create your own configuration file. See Configure JGroups (Library Mode) for instructions on how to set up JBoss Data Grid to use custom JGroups configurations in library mode.
  • In Remote Client-Server mode, the JGroups configuration is part of the main server configuration file.

When defining both the SYM_ENCRYPT and ASYM_ENCRYPT protocols, place them directly under NAKACK2 in the configuration file.

29.3.3. SYM_ENCRYPT: Using a Key Store

SYM_ENCRYPT uses store type JCEKS. To generate a keystore compatible with JCEKS, use the following command line options to keytool:

$ keytool -genseckey -alias myKey -keypass changeit -storepass changeit -keyalg Blowfish -keysize 56 -keystore defaultStore.keystore -storetype JCEKS

SYM_ENCRYPT can then be configured by adding the following information to the JGroups file used by the application.

<SYM_ENCRYPT sym_algorithm="AES"

The defaultStore.keystore must be found in the classpath.

29.3.4. ASYM_ENCRYPT: Configured with Algorithms and Key Sizes

In this encryption mode, the coordinator selects the secretKey and distributes it to all peers. There is no keystore, and keys are distributed using a public/private key exchange. Instead, encryption occurs as follows:

  1. The secret key is generated and distributed by the coordinator.
  2. When a view change occurs, a peer requests the secret key by sending a key request with its own public key.
  3. The coordinator encrypts the secret key with the public key, and sends it back to the peer.
  4. The peer then decrypts and installs the key as its own secret key.
  5. Any further communications are encrypted and decrypted using the secret key.


    <ASYM_ENCRYPT encrypt_entire_message="true"

    <AUTH auth_class="org.jgroups.auth.MD5Token"
    <pbcast.GMS join_timeout="2000" />

In the provided example, ASYM_ENCRYPT has been placed immediately below NAKACK2, and encrypt_entire_message has been enabled, indicating that the message headers will be encrypted along with the message body. This means that the NAKACK2 and UNICAST3 protocols are also encrypted. In addition, AUTH has been included as part of the configuration, so that only authenticated nodes may request the secret key from the coordinator.

View changes that identify a new controller result in a new secret key being generated and distributed to all peers. This is a substantial overhead in an application with high peer churn. A new secret key may optionally be generated when a cluster member leaves by setting change_key_on_leave to true.

When encrypting an entire message, the message must be marshalled into a byte buffer before being encrypted, resulting in decreased performance.

29.3.5. JGroups Encryption Configuration Parameters

The following table provides configuration parameters for the ENCRYPT JGroups protocol, which both SYM_ENCRYPT and ASYM_ENCRYPT extend:

Table 29.1. ENCRYPT Configuration Parameters



Cipher engine transformation for asymmetric algorithm. Default is RSA.


Initial public/private key length. Default is 512.


Cryptographic Service Provider. Default is Bouncy Castle Provider.


By default only the message body is encrypted. Enabling encrypt_entire_message ensures that all headers, destination and source addresses, and the message body is encrypted.


Cipher engine transformation for symmetric algorithm. Default is AES.


Initial key length for matching symmetric algorithm. Default is 128.


Cryptographic Service Provider. Default is Bouncy Castle Provider.

The following table provides a list of the SYM_ENCRYPT protocol parameters

Table 29.2. SYM_ENCRYPT Configuration Parameters



Alias used for recovering the key. Change the default.


Password for recovering the key. Change the default.


File on classpath that contains keystore repository.


Password used to check the integrity/unlock the keystore. Change the default.

The following table provides a list of the ASYM_ENCRYPT protocol parameters

Table 29.3. ASYM_ENCRYPT Configuration Parameters



When a member leaves the view, change the secret key, preventing old members from eavesdropping.