Chapter 29. Security for Cluster Traffic

Security can be enabled at node level via SASL protocol, which enables node authentication against a security realm. This requires nodes to authenticate each other when joining or merging with a cluster. For detailed information about security realms, see About Security Realms.

The following example depicts the <sasl /> element, which leverages the SASL protocol. Both DIGEST-MD5 or GSSAPI mechanisms are currently supported.

<management>
    <security-realms>
        <!-- Additional configuration information here -->
        <security-realm name="ClusterRealm">
            <authentication>
                <properties path="cluster-users.properties" relative-to="jboss.server.config.dir"/>
                </authentication>
                <authorization>
                    <properties path="cluster-roles.properties" relative-to="jboss.server.config.dir"/>
                </authorization>
            </security-realm>
        </security-realms>
        <!-- Additional configuration information here -->
    </security-realms>
</management>

<stack name="udp">
    <!-- Additional configuration information here -->
    <sasl mech="DIGEST-MD5" security-realm="ClusterRealm" cluster-role="cluster">
        <property name="client_name">node1</property>
        <property name="client_password">password</property>
    </sasl>
    <!-- Additional configuration information here -->
</stack>

In the provided example, the nodes use the DIGEST-MD5 mechanism to authenticate against the ClusterRealm. In order to join, nodes must have the cluster role.

The cluster-role attribute determines the role all nodes must belong to in the security realm in order to JOIN or MERGE with the cluster. Unless it has been specified, the cluster-role attribute is the name of the clustered <cache-container> by default. Each node identifies itself using the client-name property. If none is specified, the hostname on which the server is running will be used.

This name can also be overridden by specifying the jboss.node.name system property that can be overridden on the command line. For example:

$ standalone.sh -Djboss.node.name=node001

The following example demonstrates how to use DIGEST-MD5 with a properties-based security realm, with a dedicated realm for cluster node.

<management>
         <security-realms>
             <security-realm name="ClusterRealm">
                 <authentication>
                     <properties path="cluster-users.properties" relative-to="jboss.server.config.dir"/>
                 </authentication>
                 <authorization>
                     <properties path="cluster-roles.properties" relative-to="jboss.server.config.dir"/>
                 </authorization>
             </security-realm>
         </security-realms>
</management>
<subsystem xmlns="urn:infinispan:server:jgroups:8.0" default-stack="${jboss.default.jgroups.stack:udp}">
     <stack name="udp">
         <transport type="UDP" socket-binding="jgroups-udp"/>
         <protocol type="PING"/>
         <protocol type="MERGE2"/>
         <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
         <protocol type="FD_ALL"/>
         <protocol type="pbcast.NAKACK"/>
         <protocol type="UNICAST2"/>
         <protocol type="pbcast.STABLE"/>
         <protocol type="pbcast.GMS"/>
         <protocol type="UFC"/>
         <protocol type="MFC"/>
         <protocol type="FRAG3"/>
         <protocol type="RSVP"/>
         <sasl security-realm="ClusterRealm" mech="DIGEST-MD5">
             <property name="client_password>...</property>
         </sasl>
     </stack>
</subsystem>
<subsystem xmlns="urn:infinispan:server:core:8.4" default-cache-container="clustered">
     <cache-container name="clustered" default-cache="default">
         <transport executor="infinispan-transport" lock-timeout="60000" stack="udp"/>
         <!-- various clustered cache definitions here -->
     </cache-container>
</subsystem>

In the provided example, supposing the hostnames of the various nodes are node001, node002, node003, the cluster-users.properties will contain:

The cluster-roles.properties will contain:

To generate these values, the following add-users.sh script can be used:

$ add-user.sh -up cluster-users.properties -gp cluster-roles.properties -r ClusterRealm -u node001 -g clustered -p <password>

The MD5 password hash of the node must also be placed in the "client_password" property of the <sasl/> element.

<property name="client_password>...</property>

Once node security has been set up as discussed here, the cluster coordinator will validate each JOINing and MERGEing node’s credentials against the realm before letting the node become part of the cluster view.

When using the GSSAPI mechanism, the client_name is used as the name of a Kerberos-enabled login module defined within the security domain subsystem. For a full procedure on how to do this, see Configure Hot Rod Authentication (GSSAPI/Kerberos).

<security-domain name="krb-node0" cache-type="default">
    <authentication>
        <login-module code="Kerberos" flag="required">
            <module-option name="storeKey" value="true"/>
            <module-option name="useKeyTab" value="true"/>
            <module-option name="refreshKrb5Config" value="true"/>
            <module-option name="principal" value="jgroups/node0/clustered@INFINISPAN.ORG"/>
            <module-option name="keyTab" value="${jboss.server.config.dir}/keytabs/jgroups_node0_clustered.keytab"/>
            <module-option name="doNotPrompt" value="true"/>
        </login-module>
    </authentication>
</security-domain>

The following property must be set in the <sasl/> element to reference it:

<sasl <!-- Additional configuration information here --> >
     <property name="login_module_name">
     		<!-- Additional configuration information here -->
     </property>
</sasl>

As a result, the authentication section of the security realm is ignored, as the nodes will be validated against the Kerberos Domain Controller. The authorization configuration is still required, as the node principal must belong to the required cluster-role.

In all cases, it is recommended that a shared authorization database, such as LDAP, be used to validate node membership in order to simplify administration.

By default, the principal of the joining node must be in the following format:

jgroups/$NODE_NAME/$CACHE_CONTAINER_NAME@REALM

In Library mode, node authentication is configured directly in the JGroups configuration. JGroups can be configured so that nodes must authenticate each other when joining or merging with a cluster. The authentication uses SASL and is enabled by adding the SASL protocol to your JGroups XML configuration.

SASL relies on JAAS notions, such as CallbackHandlers, to obtain certain information necessary for the authentication handshake. Users must supply their own CallbackHandlers on both client and server sides.

<SASL mech="DIGEST-MD5"
    client_name="node_user"
    client_password="node_password"
    server_callback_handler_class="org.example.infinispan.security.JGroupsSaslServerCallbackHandler"
    client_callback_handler_class="org.example.infinispan.security.JGroupsSaslClientCallbackHandler"
    sasl_props="com.sun.security.sasl.digest.realm=test_realm" />

The above example uses the DIGEST-MD5 mechanism. Each node must declare the user and password it will use when joining the cluster.

For instances where a more complex Kerberos or LDAP approach is not needed the SimpleAuthorizingCallbackHandler class may be used. To enable this set both the server_callback_handler and the client_callback_handler to org.jgroups.auth.sasl.SimpleAuthorizingCallbackHandler , as seen in the below example:

<SASL mech="DIGEST-MD5"
  client_name="node_user"
  client_password="node_password"
  server_callback_handler_class="org.jgroups.auth.sasl.SimpleAuthorizingCallbackHandler"
  client_callback_handler_class="org.jgroups.auth.sasl.SimpleAuthorizingCallbackHandler"
  sasl_props="com.sun.security.sasl.digest.realm=test_realm" />

The SimpleAuthorizingCallbackHandler may be configured either programmatically, by passing the constructor an instance of of java.util.Properties , or via standard Java system properties, set on the command line using the -DpropertyName=propertyValue notation. The following properties are available:

The behavior of a node differs depending on whether it is the coordinator node or any other node. The coordinator acts as the SASL server, with the joining or merging nodes behaving as SASL clients. When using the DIGEST-MD5 mechanism in Library mode, the server and client callback must be specified so that the server and client are aware of how to obtain the credentials. Therefore, two CallbackHandlers are required:

The following example demonstrates these CallbackHandlers.

<SASL mech="DIGEST-MD5"
      client_name="node_name"
      client_password="node_password"
      client_callback_handler_class="${CLIENT_CALLBACK_HANDLER_IN_CLASSPATH}"
      server_callback_handler_class="${SERVER_CALLBACK_HANDLER_IN_CLASSPATH}"
      sasl_props="com.sun.security.sasl.digest.realm=test_realm"
/>

JGroups is designed so that all nodes are able to act as coordinator or client depending on cluster behavior, so if the current coordinator node goes down, the next node in the succession chain will become the coordinator. Given this behavior, both server and client callback handlers must be identified within SASL for Red Hat JBoss Data Grid implementations.

When performing node authentication in Library mode using the GSSAPI mechanism, the login_module_name parameter must be specified instead of callback.

This login module is used to obtain a valid Kerberos ticket, which is used to authenticate a client to the server. The server_name must also be specified, as the client principal is constructed as jgroups/$server_name@REALM .

<SASL mech="GSSAPI"
         server_name="node0/clustered"
         login_module_name="krb-node0"
         server_callback_handler_class="org.infinispan.test.integration.security.utils.SaslPropCallbackHandler" />

On the coordinator node, the server_callback_handler_class must be specified for node authorization. This will determine if the authenticated joining node has permission to join the cluster.

JGroups includes the SYM_ENCRYPT and ASYM_ENCRYPT protocols to provide encryption for cluster traffic.

By default, both of these protocols only encrypt the message body; they do not encrypt message headers. To encrypt the entire message, including all headers, as well as destination and source addresses, the property encrypt_entire_message must be true. When defining these protocols they should be placed directly under NAKACK2.

Both protocols may be used to encrypt and decrypt communication in JGroups, and are used in the following ways:

Each message is identified as encrypted with a specific encryption header identifying the encrypt header and an MD5 digest identifying the version of the key being used to encrypt and decrypt messages.

JGroups encryption protocols are placed in the JGroups configuration file, and there are three methods of including this file depending on how JBoss Data Grid is in use:

When defining both the SYM_ENCRYPT and ASYM_ENCRYPT protocols, place them directly under NAKACK2 in the configuration file.

SYM_ENCRYPT uses store type JCEKS. To generate a keystore compatible with JCEKS, use the following command line options to keytool:

$ keytool -genseckey -alias myKey -keypass changeit -storepass changeit -keyalg Blowfish -keysize 56 -keystore defaultStore.keystore -storetype JCEKS

SYM_ENCRYPT can then be configured by adding the following information to the JGroups file used by the application.

<SYM_ENCRYPT sym_algorithm="AES"
            encrypt_entire_message="true"
            keystore_name="defaultStore.keystore"
            store_password="changeit"
            alias="myKey"/>

In this encryption mode, the coordinator selects the secretKey and distributes it to all peers. There is no keystore, and keys are distributed using a public/private key exchange. Instead, encryption occurs as follows:

    ...
    <VERIFY_SUSPECT/>
    <ASYM_ENCRYPT encrypt_entire_message="true"
             sym_keylength="128"
             sym_algorithm="AES/ECB/PKCS5Padding"
             asym_keylength="512"
             asym_algorithm="RSA"/>

    <pbcast.NAKACK2/>
    <UNICAST3/>
    <pbcast.STABLE/>
    <FRAG3/>
    <AUTH auth_class="org.jgroups.auth.MD5Token"
          auth_value="chris"
          token_hash="MD5"/>
    <pbcast.GMS join_timeout="2000" />

In the provided example, ASYM_ENCRYPT has been placed immediately below NAKACK2, and encrypt_entire_message has been enabled, indicating that the message headers will be encrypted along with the message body. This means that the NAKACK2 and UNICAST3 protocols are also encrypted. In addition, AUTH has been included as part of the configuration, so that only authenticated nodes may request the secret key from the coordinator.

View changes that identify a new controller result in a new secret key being generated and distributed to all peers. This is a substantial overhead in an application with high peer churn. A new secret key may optionally be generated when a cluster member leaves by setting change_key_on_leave to true.

When encrypting an entire message, the message must be marshalled into a byte buffer before being encrypted, resulting in decreased performance.

The following table provides configuration parameters for the ENCRYPT JGroups protocol, which both SYM_ENCRYPT and ASYM_ENCRYPT extend:

The following table provides a list of the SYM_ENCRYPT protocol parameters

The following table provides a list of the ASYM_ENCRYPT protocol parameters