When you bring Data Grid clusters back online after shutting them down, you should wait for the cluster to be available before adding or removing nodes or modifying cluster state.

If you shutdown clustered nodes with the shutdown server command, you must restart each server in reverse order.
For example, if you shutdown server1 and then shutdown server2, you should first start server2 and then start server1.

If you shutdown a cluster with the shutdown cluster command, clusters become fully operational only after all nodes rejoin.
You can restart nodes in any order but the cluster remains in DEGRADED state until all nodes that were joined before shutdown are running.

Apart from resources in the bin and docs folders, the only folder under $RHDG_HOME that you should interact with is the server root directory, which is named server by default.

You can create multiple nodes under the same $RHDG_HOME directory or in different directories, but each Data Grid Server instance must have its own server root directory. For example, a cluster of 5 nodes could have the following server root directories on the filesystem:

├── server
├── server1
├── server2
├── server3
└── server4

Each server root directory should contain the following folders:

├── server
│   ├── conf
│   ├── data
│   ├── lib
│   └── log

server/conf

Holds infinispan.xml configuration files for a Data Grid Server instance.

Data Grid separates configuration into two layers:

server/data

Provides internal storage that Data Grid Server uses to maintain cluster state.

server/lib

Contains extension JAR files for custom filters, custom event listeners, JDBC drivers, custom ServerTask implementations, and so on.

server/log

Holds Data Grid Server log files.

Uses an inet-address strategy that maps a single public interface to the IPv4 loopback address (127.0.0.1).

<interfaces>
  <interface name="public">
    <inet-address value="${infinispan.bind.address:127.0.0.1}"/>
  </interface>
</interfaces>

Selects a loopback address.

<interfaces>
    <interface name="public">
        <loopback/>
    </interface>
</interfaces>

Selects a non-loopback address.

<interfaces>
    <interface name="public">
        <non-loopback/>
    </interface>
</interfaces>

Selects networks based on IP address.

<interfaces>
    <interface name="public">
        <inet-address value="10.1.2.3"/>
    </interface>
</interfaces>

Selects the INADDR_ANY wildcard address. As a result Data Grid servers listen on all interfaces.

<interfaces>
    <interface name="public">
        <any-address/>
    </interface>
</interfaces>

Selects a link-local IP address.

<interfaces>
    <interface name="public">
        <inet-address value="10.1.2.3"/>
    </interface>
</interfaces>

Selects a site-local (private) IP address.

<interfaces>
    <interface name="public">
        <inet-address value="10.1.2.3"/>
    </interface>
</interfaces>

Resolves the host name and selects one of the IP addresses that is assigned to any network interface.

Data Grid servers enumerate all available operating system interfaces to locate IP addresses resolved from the host name in your configuration.

<interfaces>
    <interface name="public">
        <match-host value="my_host_name"/>
    </interface>
</interfaces>

Selects an IP address assigned to a network interface that matches a regular expression.

Data Grid servers enumerate all available operating system interfaces to locate the interface name in your configuration.

<interfaces>
    <interface name="public">
        <match-interface value="eth0"/>
    </interface>
</interfaces>

Similar to inet-address but selects an IP address using a regular expression.

Data Grid servers enumerate all available operating system interfaces to locate the IP address in your configuration.

<interfaces>
    <interface name="public">
        <match-address value="132\..*"/>
    </interface>
</interfaces>

Interface configurations can include multiple strategies. Data Grid servers try each strategy in the declared order.

For example, with the following configuration, Data Grid servers first attempt to match a host, then an IP address, and then fall back to the INADDR_ANY wildcard address:

<interfaces>
    <interface name="public">
        <match-host value="my_host_name"/>
        <match-address value="132\..*"/>
        <any-address/>
    </interface>
</interfaces>

You can use the server -b switch or the infinispan.bind.address system property to bind to a different address.

For example, bind the public interface to 127.0.0.2 as follows:

$ bin/server.sh -b 127.0.0.2

bin\server.bat -b 127.0.0.2

Configure port offsets with Data Grid servers when running multiple instances on the same host. The default port offset is 0.

Use the -o switch with the Data Grid CLI or the infinispan.socket.binding.port-offset system property to set port offsets.

For example, start a server instance with an offset of 100 as follows. With the default configuration, this results in the Data Grid server listening on port 11322.

$ bin/server.sh -o 100

bin\server.bat -o 100

Configure clients to provide ALPN messages for protocol negotiation during TLS handshakes with Data Grid servers.

ConfigurationBuilder builder = new ConfigurationBuilder()
      .addServers("127.0.0.1:11222");

builder.security().ssl().enable()
      .trustStoreFileName("truststore.pkcs12")
      .trustStorePassword(DEFAULT_TRUSTSTORE_PASSWORD.toCharArray());

RemoteCacheManager remoteCacheManager = new RemoteCacheManager(builder.build());
RemoteCache<String, String> cache = remoteCacheManager.getCache("default"");

infinispan.client.hotrod.server_list = 127.0.0.1:11222
infinispan.client.hotrod.use_ssl = true
infinispan.client.hotrod.trust_store_file_name = truststore.pkcs12
infinispan.client.hotrod.trust_store_password = trust_store_password

Hot Rod is a binary TCP client-server protocol designed to provide faster data access and improved performance in comparison to text-based protocols.

Data Grid provides Hot Rod client libraries in Java, C++, C#, Node.js and other programming languages.

When client send requests, Data Grid servers compare the topology ID in request headers with the topology ID from the cache. Data Grid servers send new topology views if client have older topology IDs.

Cluster topology views allow Hot Rod clients to immediately detect when nodes join and leave, which enables dynamic load balancing and failover.

In distributed cache modes, the consistent hashing algorithm also makes it possible to route Hot Rod client requests directly to primary owners.

You can use standard HTTP load balancers to provide clients with load balancing and failover capabilities. However, HTTP load balancers maintain static cluster views and require manual updates when cluster topology changes occur.

Hot Rod connector declarations enable Hot Rod servers.

<hotrod-connector name="hotrod">
  <topology-state-transfer />
  <authentication>
    ...
  </authentication>
  <encryption>
    ...
  </encryption>
</hotrod-connector>

REST connector declarations enable REST servers.

<rest-connector name="rest">
  <authentication>
    ...
  </authentication>
  <cors-rules>
    ...
  </cors-rules>
  <encryption>
    ...
  </encryption>
</rest-connector>

Adjust any firewall rules to allow traffic between the server and external clients.

# firewall-cmd --add-port=11222/tcp --permanent
success
# firewall-cmd --list-ports | grep 11222
11222/tcp

To configure firewall rules that apply across a network, you can use the nftables utility.

Property realms use property files to define users and groups.

users.properties maps usernames to passwords in plain-text format. Passwords can also be pre-digested if you use the DIGEST-MD5 SASL mechanism or Digest HTTP mechanism.

myuser=a_password
user2=another_password

groups.properties maps users to roles.

myuser=supervisor,reader,writer
user2=supervisor

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:11.0 https://infinispan.org/schemas/infinispan-server-11.0.xsd"
          xmlns="urn:infinispan:server:11.0">
   <security-realms>
      <security-realm name="default">
         <properties-realm groups-attribute="Roles"> 1
            <user-properties path="users.properties" 2
                             relative-to="infinispan.server.config.path" 3
                             plain-text="true"/> 4
            <group-properties path="groups.properties" 5
                              relative-to="infinispan.server.config.path"/>
         </properties-realm>
      </security-realm>
   </security-realms>
</security>

LDAP realms connect to LDAP servers, such as OpenLDAP, Red Hat Directory Server, Apache Directory Server, or Microsoft Active Directory, to authenticate users and obtain membership information.

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:11.0 https://infinispan.org/schemas/infinispan-server-11.0.xsd"
          xmlns="urn:infinispan:server:11.0">
   <security-realms>
      <security-realm name="default">
        <ldap-realm name="ldap" 1
                    url="ldap://my-ldap-server:10389" 2
                    principal="uid=admin,ou=People,dc=infinispan,dc=org" 3
                    credential="strongPassword"
                    connection-timeout="3000" read-timeout="30000" 4
                    connection-pooling="true" referral-mode="ignore"
                    page-size="30"
                    direct-verification="true"> 5
            <identity-mapping rdn-identifier="uid" 6
                              search-dn="ou=People,dc=infinispan,dc=org"> 7
               <attribute-mapping> 8
                  <attribute from="cn"
                             to="Roles"
                             filter="(&amp;(objectClass=groupOfNames)(member={1}))"
                             filter-dn="ou=Roles,dc=infinispan,dc=org"/>
               </attribute-mapping>
            </identity-mapping>
         </ldap-realm>
      </security-realm>
   </security-realms>
</security>

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:11.0 https://infinispan.org/schemas/infinispan-server-11.0.xsd"
          xmlns="urn:infinispan:server:11.0">
   <security-realms>
      <security-realm name="default">
         <ldap-realm name="ldap"
                     url="ldap://${org.infinispan.test.host.address}:10389"
                     principal="uid=admin,ou=People,dc=infinispan,dc=org"
                     credential="strongPassword">
            <name-rewriter> 1
               <regex-principal-transformer name="domain-remover"
                                            pattern="(.*)@INFINISPAN\.ORG"
                                            replacement="$1"/>
            </name-rewriter>
            <identity-mapping rdn-identifier="uid"
                              search-dn="ou=People,dc=infinispan,dc=org">
               <attribute-mapping>
                  <attribute from="cn" to="Roles"
                             filter="(&amp;(objectClass=groupOfNames)(member={1}))"
                             filter-dn="ou=Roles,dc=infinispan,dc=org" />
               </attribute-mapping>
               <user-password-mapper from="userPassword" />
            </identity-mapping>
         </ldap-realm>
      </security-realm>
   </security-realms>
</security>

Trust store realms use keystores that contain the public certificates of all clients that are allowed to connect to Data Grid server.

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:11.0 https://infinispan.org/schemas/infinispan-server-11.0.xsd"
          xmlns="urn:infinispan:server:11.0">
   <security-realms>
      <security-realm name="default">
         <server-identities>
            <ssl>
               <keystore path="server.p12" 1
                         relative-to="infinispan.server.config.path" 2
                         keystore-password="secret" 3
                         alias="server"/> 4
            </ssl>
         </server-identities>
         <truststore-realm path="trust.p12" 5
                           relative-to="infinispan.server.config.path"
                           keystore-password="secret"/>
      </security-realm>
   </security-realms>
</security>

Token realms use external services to validate tokens and require providers that are compatible with RFC-7662 (OAuth2 Token Introspection), such as Red Hat SSO.

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:11.0 https://infinispan.org/schemas/infinispan-server-11.0.xsd"
          xmlns="urn:infinispan:server:11.0">
   <security-realms>
      <security-realm name="default">
         <token-realm name="token"
                      auth-server-url="https://oauth-server/auth/"> 1
            <oauth2-introspection
                    introspection-url="https://oauth-server/auth/realms/infinispan/protocol/openid-connect/token/introspect" 2
                    client-id="infinispan-server" 3
                    client-secret="1fdca4ec-c416-47e0-867a-3d471af7050f"/> 4
         </token-realm>
      </security-realm>
   </security-realms>
</security>

SSL identities use keystores that contain either a certificate or chain of certificates.

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:11.0 https://infinispan.org/schemas/infinispan-server-11.0.xsd"
          xmlns="urn:infinispan:server:11.0">
   <security-realms>
      <security-realm name="default">
         <server-identities> 1
            <ssl> 2
               <keystore path="server.p12" 3
                         relative-to="infinispan.server.config.path" 4
                         keystore-password="secret" 5
                         alias="server"/> 6
            </ssl>
         </server-identities>
      </security-realm>
   </security-realms>
</security>

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:11.0 https://infinispan.org/schemas/infinispan-server-11.0.xsd"
          xmlns="urn:infinispan:server:11.0">
   <security-realms>
      <security-realm name="default">
         <server-identities>
            <ssl>
               <keystore path="server.p12"
                         relative-to="infinispan.server.config.path"
                         keystore-password="secret"
                         alias="server"
                         generate-self-signed-certificate-host="localhost"/> 1
            </ssl>
         </server-identities>
      </security-realm>
   </security-realms>
</security>

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:11.0
          https://infinispan.org/schemas/infinispan-server-11.0.xsd"
          xmlns="urn:infinispan:server:11.0">
   <security-realms>
      <security-realm name="default">
         <server-identities>
            <ssl>
               <keystore path="server.p12"
                         relative-to="infinispan.server.config.path"
                         keystore-password="secret" alias="server"/>
               <engine enabled-protocols="TLSv1.2 TLSv1.1" 1
                       enabled-ciphersuites="SSL_RSA_WITH_AES_128_GCM_SHA256 2
                       SSL_RSA_WITH_AES_128_CBC_SHA256"/>
            </ssl>
         </server-identities>
      </security-realm>
   </security-realms>
</security>

Kerberos identities use keytab files that contain service principal names and encrypted keys, derived from Kerberos passwords.

In most cases, you create unique principals for the Hot Rod and REST connectors. For example, you have a "datagrid" server in the "INFINISPAN.ORG" domain. In this case you should create the following service principals:

<security xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:server:11.0 https://infinispan.org/schemas/infinispan-server-11.0.xsd"
          xmlns="urn:infinispan:server:11.0">
   <security-realms>
      <security-realm name="default">
         <server-identities> 1
            <kerberos keytab-path="hotrod.keytab" 2
                      principal="hotrod/datagrid@INFINISPAN.ORG" 3
                      required="true"/> 4
            <kerberos keytab-path="http.keytab" 5
                      principal="HTTP/localhost@INFINISPAN.ORG" 6
                      required="true"/>
         </server-identities>
      </security-realm>
   </security-realms>
</security>

Data Grid servers automatically configure authentication mechanisms based on the security realm that you assign to endpoints.

<security-realm name="default">
  <properties-realm groups-attribute="Roles">
    <user-properties path="users.properties"
                     relative-to="infinispan.server.config.path"
                     plain-text="true"/>
    <group-properties path="groups.properties"
                      relative-to="infinispan.server.config.path" />
  </properties-realm>
</security-realm>

The endpoints configuration assigns the "default" security realm to the Hot Rod and REST connectors, as follows:

<endpoints socket-binding="default" security-realm="default">
  <hotrod-connector name="hotrod"/>
  <rest-connector name="rest"/>
</endpoints>

As a result of the preceding configuration, Data Grid servers require authentication with a mechanism that the property realm supports.

Explicitly configure Hot Rod connector authentication to override the default SASL authentication mechanisms that Data Grid servers use for security realms.

<endpoints xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="urn:infinispan:server:11.0
           https://infinispan.org/schemas/infinispan-server-11.0.xsd"
           xmlns="urn:infinispan:server:11.0"
           socket-binding="default" security-realm="default"> 1
   <hotrod-connector name="hotrod">
      <authentication>
         <sasl mechanisms="SCRAM-SHA-512 SCRAM-SHA-384 SCRAM-SHA-256 2
                           SCRAM-SHA-1 DIGEST-SHA-512 DIGEST-SHA-384
                           DIGEST-SHA-256 DIGEST-SHA DIGEST-MD5 PLAIN"
               server-name="infinispan" 3
               qop="auth"/> 4
      </authentication>
   </hotrod-connector>
</endpoints>

<endpoints xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="urn:infinispan:server:11.0 https://infinispan.org/schemas/infinispan-server-11.0.xsd"
           xmlns="urn:infinispan:server:11.0"
           socket-binding="default" security-realm="default">
   <hotrod-connector name="hotrod">
      <authentication>
         <sasl mechanisms="GSSAPI GS2-KRB5" 1
               server-name="datagrid" 2
               server-principal="hotrod/datagrid@INFINISPAN.ORG"/> 3
      </authentication>
   </hotrod-connector>
</endpoints>

<hotrod-connector socket-binding="hotrod" cache-container="default">
   <authentication security-realm="ApplicationRealm">
      <sasl server-name="myhotrodserver"
            mechanisms="PLAIN DIGEST-MD5 GSSAPI EXTERNAL" 1
            qop="auth">
         <policy> 2
            <no-active value="true" />
            <no-anonymous value="true" />
            <no-plain-text value="true" />
         </policy>
      </sasl>
   </authentication>
</hotrod-connector>

Explicitly configure REST connector authentication to override the default HTTP authentication mechanisms that Data Grid servers use for security realms.

<endpoints xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="urn:infinispan:server:11.0 https://infinispan.org/schemas/infinispan-server-11.0.xsd"
           xmlns="urn:infinispan:server:11.0"
           socket-binding="default" security-realm="default"> 1
   <rest-connector name="rest">
      <authentication mechanisms="DIGEST BASIC"/> 2
   </rest-connector>
</endpoints>

<endpoints xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="urn:infinispan:server:11.0 https://infinispan.org/schemas/infinispan-server-11.0.xsd"
           xmlns="urn:infinispan:server:11.0"
           socket-binding="default" security-realm="default">
   <rest-connector name="rest">
      <authentication mechanisms="SPNEGO" 1
                      server-principal="HTTP/localhost@INFINISPAN.ORG"/> 2
   </rest-connector>
</endpoints>

Data Grid lets you configure authorization to secure Cache Managers and cache instances. When user applications or clients attempt to perform an operation on secured Cached Managers and caches, they must provide an identity with a role that has sufficient permissions to perform that operation.

For example, you configure authorization on a specific cache instance so that invoking Cache.get() requires an identity to be assigned a role with read permission while Cache.put() requires a role with write permission.

In this scenario, if a user application or client with the reader role attempts to write an entry, Data Grid denies the request and throws a security exception. If a user application or client with the writer role sends a write request, Data Grid validates authorization and issues a token for subsequent operations.

Data Grid uses role mappers so that security principals correspond to roles, which represent one or more permissions.

The following image illustrates how security principals map to roles:

<role name="standard" permission="READ WRITE" />
<role name="supervisors" permission="READ WRITE EXEC BULK"/>

Configure authorization in your infinispan.xml file.

Learn about default JGroups stacks that configure cluster transport.

Data Grid uses the following ports for cluster transport messages:

Cross-Site Replication

Data Grid uses the following ports for the JGroups RELAY2 protocol:

When you extend a JGroups stack, inheritance attributes let you adjust protocols and properties in the stack you are extending.

Set system properties that configure JGroups cluster transport stacks.

PING, or UDPPING is a generic JGroups discovery mechanism that uses dynamic multicasting with the UDP protocol.

When joining, nodes send PING requests to an IP multicast address to discover other nodes already in the Data Grid cluster. Each node responds to the PING request with a packet that contains the address of the coordinator node and its own address. C=coordinator’s address and A=own address. If no nodes respond to the PING request, the joining node becomes the coordinator node in a new cluster.

<config>
  <PING num_discovery_runs="3"/>
  ...
</config>

TCPPING is a generic JGroups discovery mechanism that uses a list of static addresses for cluster members.

With TCPPING, you manually specify the IP address or hostname of each node in the Data Grid cluster as part of the JGroups stack, rather than letting nodes discover each other dynamically.

<config>
  <TCP bind_port="7800" />
  <TCPPING timeout="3000"
           initial_hosts="${jgroups.tcpping.initial_hosts:hostname1[port1],hostname2[port2]}"
          port_range="0" 1
          num_initial_members="3"/>
  ...
</config>

MPING uses IP multicast to discover the initial membership of Data Grid clusters.

You can use MPING to replace TCPPING discovery with TCP stacks and use multicasing for discovery instead of static lists of initial hosts. However, you can also use MPING with UDP stacks.

<config>
  <MPING mcast_addr="${jgroups.mcast_addr:228.6.7.8}"
         mcast_port="${jgroups.mcast_port:46655}"
         num_discovery_runs="3"
         ip_ttl="${jgroups.udp.ip_ttl:2}"/>
  ...
</config>

Gossip routers provide a centralized location on the network from which your Data Grid cluster can retrieve addresses of other nodes.

You inject the address (IP:PORT) of the Gossip router into Data Grid nodes as follows:

<config>
  <TCP bind_port="7800" />
  <TCPGOSSIP timeout="3000"
             initial_hosts="${GossipRouterAddress}"
             num_initial_members="3" />
  ...
</config>

JDBC_PING uses shared databases to store information about Data Grid clusters. This protocol supports any database that can use a JDBC connection.

Nodes write their IP addresses to the shared database so joining nodes can find the Data Grid cluster on the network. When nodes leave Data Grid clusters, they delete their IP addresses from the shared database.

<config>
  <JDBC_PING connection_url="jdbc:mysql://localhost:3306/database_name"
             connection_username="user"
             connection_password="password"
             connection_driver="com.mysql.jdbc.Driver"/>
  ...
</config>

JGroups DNS_PING queries DNS servers to discover Data Grid cluster members in Kubernetes environments such as OKD and Red Hat OpenShift.

<config>
  <dns.DNS_PING dns_query="myservice.myproject.svc.cluster.local" />
  ...
</config>

To secure cluster traffic, you configure Data Grid nodes to encrypt JGroups message payloads with secret keys.

Data Grid nodes can obtain secret keys from either:

Asymmetric encryption secures cluster traffic as follows:

SYM_ENCRYPT, on the other hand, is faster than ASYM_ENCRYPT because nodes do not need to exchange keys with the cluster coordinator. A potential drawback to SYM_ENCRYPT is that there is no configuration to automatically generate new secret keys when cluster membership changes. Users are responsible for generating and distributing the secret keys that nodes use to encrypt cluster traffic.

Configure Data Grid clusters to generate and distribute secret keys that encrypt JGroups messages.

[org.infinispan.CLUSTER] ISPN000078: Starting JGroups channel cluster with stack <encrypted_stack_name>

Data Grid nodes can join the cluster only if they use ASYM_ENCRYPT and can obtain the secret key from the coordinator node. Otherwise the following message is written to Data Grid logs:

[org.jgroups.protocols.ASYM_ENCRYPT] <hostname>: received message without encrypt header from <hostname>; dropping it

Configure Data Grid clusters to encrypt JGroups messages with secret keys from keystores that you provide.

[org.infinispan.CLUSTER] ISPN000078: Starting JGroups channel cluster with stack <encrypted_stack_name>

Data Grid nodes can join the cluster only if they use SYM_ENCRYPT and can obtain the secret key from the shared keystore. Otherwise the following message is written to Data Grid logs:

[org.jgroups.protocols.SYM_ENCRYPT] <hostname>: received message without encrypt header from <hostname>; dropping it

Data Grid configuration in XML format must conform to the schema and include:

<infinispan>
    <cache-container>
        <distributed-cache name="myCache" mode="SYNC">
          <encoding media-type="application/x-protostream"/>
          <memory max-count="1000000" when-full="REMOVE"/>
        </distributed-cache>
    </cache-container>
</infinispan>

Data Grid configuration in JSON format:

{
  "distributed-cache": {
    "name": "myCache",
    "mode": "SYNC",
    "encoding": {
      "media-type": "application/x-protostream"
      },
    "memory": {
      "max-count": 1000000,
      "when-full": "REMOVE"
    }
  }
}

Data Grid server tasks are classes that extend the org.infinispan.tasks.ServerTask interface and generally include the following method calls:

The following HelloTask class implementation provides an example task that has one parameter:

package example;

import org.infinispan.tasks.ServerTask;
import org.infinispan.tasks.TaskContext;

public class HelloTask implements ServerTask<String> {

   private TaskContext ctx;

   @Override
   public void setTaskContext(TaskContext ctx) {
      this.ctx = ctx;
   }

   @Override
   public String call() throws Exception {
      String name = (String) ctx.getParameters().get().get("name");
      return "Hello " + name;
   }

   @Override
   public String getName() {
      return "hello-task";
   }

}

Add your custom server task classes to Data Grid servers.

Data Grid server scripting is based on the javax.script API and is compatible with any JVM-based ScriptEngine implementation.

// mode=local,language=javascript,parameters=[greetee]
"Hello " + greetee

When you run the preceding script, you pass a value for the greetee parameter and Data Grid returns "Hello ${value}".

// name=test, language=javascript
// mode=local, parameters=[a,b,c]

// mode=local,language=javascript
multiplicand * multiplier

Use the command line interface to add scripts to Data Grid servers.

Add scripts with the Hot Rod RemoteCache interface as in the following example:

RemoteCache<String, String> scriptCache = cacheManager.getCache("___script_cache");
scriptCache.put("multiplication.js",
  "// mode=local,language=javascript\n" +
  "multiplicand * multiplier\n");

Use the command line interface to run tasks and scripts on Data Grid clusters.

Call the execute() method to run scripts with the Hot Rod RemoteCache interface, as in the following example:

RemoteCache<String, Integer> cache = cacheManager.getCache();
// Create parameters for script execution.
Map<String, Object> params = new HashMap<>();
params.put("multiplicand", 10);
params.put("multiplier", 20);
// Run the script with the parameters.
Object result = cache.execute("multiplication.js", params);

Call the execute() method to run tasks with the Hot Rod RemoteCache interface, as in the following example:

// Add configuration for a locally running server.
ConfigurationBuilder builder = new ConfigurationBuilder();
builder.addServer().host("127.0.0.1").port(11222);

// Connect to the server.
RemoteCacheManager cacheManager = new RemoteCacheManager(builder.build());

// Retrieve the remote cache.
RemoteCache<String, String> cache = cacheManager.getCache();

// Create task parameters.
Map<String, String> parameters = new HashMap<>();
parameters.put("name", "developer");

// Run the server task.
String greet = cache.execute("hello-task", parameters);
System.out.println(greet);

Data Grid writes log messages to the following directory:
$RHDG_HOME/${infinispan.server.root}/log

You configure Data Grid logs with log4j2.xml, which is described in the Log4j 2 manual.

Hot Rod and REST endpoints can record all inbound client requests as log entries with the following categories:

<Logger name="org.infinispan.HOTROD_ACCESS_LOG" additivity="false" level="TRACE">
   <AppenderRef ref="HR-ACCESS-FILE"/>
</Logger>

%X{address} %X{user} [%d{dd/MMM/yyyy:HH:mm:ss Z}] "%X{method} %m
%X{protocol}" %X{status} %X{requestSize} %X{responseSize} %X{duration}%n

Data Grid lets you enable statistics for Cache Managers and caches. However, enabling statistics for a Cache Manager does not enable statistics for the caches that it controls. You must explicitly enable statistics for your caches.

<cache-container statistics="true"> 1
  <local-cache name="mycache" statistics="true"/> 2
</cache-container>

GlobalConfiguration globalConfig = new GlobalConfigurationBuilder()
  .cacheContainer().statistics(true) 1
  .build();

 ...

Configuration config = new ConfigurationBuilder()
  .statistics().enable() 2
  .build();

Configure Data Grid to export gauges and histograms.

<cache-container statistics="true"> 1
  <metrics gauges="true" histograms="true" /> 2
</cache-container>

GlobalConfiguration globalConfig = new GlobalConfigurationBuilder()
  .statistics().enable() 1
  .metrics().gauges(true).histograms(true) 2
  .build();

Collect Data Grid metrics with monitoring tools such as Prometheus.

static_configs:
    - targets: ['localhost:11222']

Data Grid can register JMX MBeans that you can use to collect statistics and perform administrative operations. However, you must enable statistics separately to JMX otherwise Data Grid provides 0 values for all statistic attributes.

<cache-container>
  <jmx enabled="true" /> 1
</cache-container>

GlobalConfiguration globalConfig = new GlobalConfigurationBuilder()
  .jmx().enable() 1
  .build();

Retrieve Data Grid cluster health statistics via JMX.

Get Data Grid cluster health via the REST API.

Data Grid responds with a JSON document such as the following:

{
    "cluster_health":{
        "cluster_name":"ISPN",
        "health_status":"HEALTHY",
        "number_of_nodes":2,
        "node_names":[
            "NodeA-36229",
            "NodeB-28703"
        ]
    },
    "cache_health":[
        {
            "status":"HEALTHY",
            "cache_name":"___protobuf_metadata"
        },
        {
            "status":"HEALTHY",
            "cache_name":"cache2"
        },
        {
            "status":"HEALTHY",
            "cache_name":"mycache"
        },
        {
            "status":"HEALTHY",
            "cache_name":"cache1"
        }
    ]

}

GET /rest/v2/cache-managers/{cacheManagerName}/health/status

You must use specific remote cache store configuration to perform rolling upgrades, as follows:

<persistence passivation="false"> 1
   <remote-store xmlns="urn:infinispan:config:store:remote:11.0"
                 cache="myDistCache" 2
                 protocol-version="2.5" 3
                 hotrod-wrapping="true" 4
                 raw-values="true" 5
                 segmented="false"> 6
      <remote-server host="127.0.0.1" port="11222"/> 7
   </remote-store>
</persistence>

Data Grid Server requires JDK 11 or later.

Use the server script to run Data Grid Server instances.

$RHDG_HOME/bin/server.sh

$RHDG_HOME\bin\server.bat

Use the shutdown command with the CLI to perform a graceful shutdown.

Alternatively, enter Ctrl-C from the terminal to interrupt the server process or kill it via the TERM signal.

Server configuration extends Data Grid configuration with the following server-specific elements:

The default configuration file is $RHDG_HOME/server/conf/infinispan.xml.

Use different configuration files with the -c argument, as in the following example that starts a server without clustering capabilities:

$RHDG_HOME/bin/server.sh -c infinispan-local.xml

$RHDG_HOME\bin\server.bat -c infinispan-local.xml

Data Grid Server binds to the loopback IP address localhost on your network by default.

Use the -b argument to set a different IP address, as in the following example that binds to all network interfaces:

$RHDG_HOME/bin/server.sh -b 0.0.0.0

$RHDG_HOME\bin\server.bat -b 0.0.0.0

Data Grid Server listens on port 11222 by default.

Use the -p argument to set an alternative port:

$RHDG_HOME/bin/server.sh -p 30000

$RHDG_HOME\bin\server.bat -p 30000

Data Grid Server configuration defines cluster transport so multiple instances on the same network discover each other and automatically form clusters.

Use the -k argument to change the IP address for cluster traffic:

$RHDG_HOME/bin/server.sh -k 192.168.1.100

$RHDG_HOME\bin\server.bat -k 192.168.1.100

JGroups stacks configure the protocols for cluster transport. Data Grid Server uses the tcp stack by default.

Use alternative cluster stacks with the -j argument, as in the following example that uses UDP for cluster transport:

$RHDG_HOME/bin/server.sh -j udp

$RHDG_HOME\bin\server.bat -j udp

Data Grid Server requires authentication.

Create a username and password with the CLI as follows:

$RHDG_HOME/bin/cli.sh user create username -p "qwer1234!"

$RHDG_HOME\bin\cli.bat user create username -p "qwer1234!"

Data Grid Server uses infinispan.server.home.path to locate the contents of the server distribution on the host filesystem.

The server home directory, referred to as $RHDG_HOME, contains the following folders:

├── bin
├── boot
├── docs
├── lib
├── server
└── static

Data Grid Server uses infinispan.server.root.path to locate configuration files and data for Data Grid Server instances.

You can create multiple server root folders in the same directory or in different directories and then specify the locations with the -s or --server-root argument, as in the following example:

$RHDG_HOME/bin/server.sh -s server2

$RHDG_HOME\bin\server.bat -s server2

Each server root directory contains the following folders:

├── server
│   ├── conf
│   ├── data
│   ├── lib
│   └── log

Configure Data Grid Server logging with the log4j2.xml file in the server/conf folder.

Use the --logging-config=<path_to_logfile> argument to use custom paths, as follows:

$RHDG_HOME/bin/server.sh --logging-config=/path/to/log4j2.xml

$RHDG_HOME\bin\server.bat --logging-config=path\to\log4j2.xml