Site masters are the nodes in Data Grid clusters that are responsible for sending and receiving requests from backup locations.

If a node is not a site master, it must forward backup requests to a local site master. Only site masters can send requests to backup locations.

For optimal performance, you should configure all nodes as site masters. This increases the speed of backup requests because each node in the cluster can backup to remote sites directly without having to forward backup requests to site masters.

When Data Grid replicates data to backup locations, it waits until the operation completes before writing to the local cache.

You can control how Data Grid handles writes to the local cache if backup operations fail. For example, you can configure Data Grid to attempt to abort local writes and throw exceptions if backups to remote sites fail.

Synchronous backups also support two-phase commits with caches that participate in optimistic transactions. The first phase of the backup acquires a lock. The second phase commits the modification.

When Data Grid replicates data to backup locations, it does not wait until the operation completes before writing to the local cache.

Asynchronous backup operations and writes to the local cache are independent of each other. If backup operations fail, write operations to the local cache continue and no exceptions occur.

Synchronous backups offer the strongest guarantee of data consistency across sites. If strategy=sync, when cache.put() calls return you know the value is up to date in the local cache and in the backup locations.

The trade-off for this consistency is performance. Synchronous backups have much greater latency in comparison to asynchronous backups.

Asynchronous backups, on the other hand, do not add latency to client requests so they have no performance impact. However, if strategy=async, when cache.put() calls return you cannot be sure of the value in the backup locations is the same as in the local cache.

Conflicting entries can occur with Active/Active site configurations if clients write to the same entries at the same time but at different sites.

For example, client A writes to "k1" in LON at the same time that client B writes to "k1" in NYC. In this case, "k1" has a different value in LON than in NYC. After replication occurs, there is no guarantee which value for "k1" exists at which site.

To ensure data consistency, Data Grid uses a vector clock algorithm to detect conflicting entries during backup operations, as in the following illustration:

            LON       NYC

k1=(n/a)    0,0       0,0

k1=2        1,0  -->  1,0   k1=2

k1=3        1,1  <--  1,1   k1=3

k1=5        2,1       1,2   k1=8

                 -->  2,1 (conflict)
(conflict)  1,2  <--

Vector clocks are timestamp metadata that increment with each write to an entry. In the preceding example, 0,0 represents the initial value for the vector clock on "k1".

A client puts "k1=2" in LON and the vector clock is 1,0, which Data Grid replicates to NYC. A client then puts "k1=3" in NYC and the vector clock updates to 1,1, which Data Grid replicates to LON.

However if a client puts "k1=5" in LON at the same time that a client puts "k1=8" in NYC, Data Grid detects a conflicting entry because the vector value for "k1" is not strictly greater or less between LON and NYC.

When it finds conflicting entries, Data Grid uses the Java compareTo(String anotherString) method to compare site names. To determine which key takes priority, Data Grid selects the site name that is lexicographically less than the other. Keys from a site named AAA take priority over keys from a site named AAB and so on.

Following the same example, to resolve the conflict for "k1", Data Grid uses the value for "k1" that originates from LON. This results in "k1=5" in both LON and NYC after Data Grid resolves the conflict and replicates the value.

Data Grid clusters use JGroups RELAY2 for inter-cluster discovery and communication.

<jgroups>
   <stack name="xsite" 1
          extends="udp"> 2
      <relay.RELAY2 xmlns="urn:org:jgroups" 3
                    site="LON" 4
                    max_site_masters="1000"/> 5
      <remote-sites default-stack="tcp"> 6
         <remote-site name="LON"/> 7
         <remote-site name="NYC"/>
      </remote-sites>
   </stack>
</jgroups>

<jgroups>
   <stack name="relay-global" extends="tcp"> 1
         <MPING stack.combine="REMOVE"/>
         <TCPPING initial_hosts="192.0.2.0[7800]" stack.combine="INSERT_AFTER" stack.position="TCP"/>
    </stack>
   <stack name="xsite" extends="udp">
      <relay.RELAY2 site="LON" xmlns="urn:org:jgroups"
                    max_site_masters="10" 2
                    can_become_site_master="true"/>
      <remote-sites default-stack="relay-global">
         <remote-site name="LON"/>
         <remote-site name="NYC"/>
      </remote-sites>
   </stack>
</jgroups>

You can also reference externally defined JGroups stack files as follows:

<stack-file name="relay-global" path="jgroups-relay.xml"/>

In the preceding configuration jgroups-relay.xml provides a JGroups stack such as this:

<config xmlns="urn:org:jgroups"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups-4.2.xsd">

    <!-- Use TCP for inter-cluster transport. -->
    <TCP bind_addr="127.0.0.1"
         bind_port="7200"
         port_range="30"

         thread_pool.min_threads="0"
         thread_pool.max_threads="8"
         thread_pool.keep_alive_time="5000"
    />

    <!-- Use TCPPING for inter-cluster discovery. -->
    <TCPPING timeout="3000"
             initial_hosts="127.0.0.1[7200]"
             port_range="3"
             ergonomics="false"/>

    <!-- Provide other JGroups stack configuration as required. -->
</config>

Take backup locations offline manually and bring them back online.

For more information and examples, run the help site command.

Transfer cache state to remote backup locations.

For more information and examples, run the help site command.

Retrieve the status of all backup locations with GET requests.

GET /v2/caches/{cacheName}/x-site/backups/

Data Grid responds with the status of each backup location in JSON format, as in the following example:

{
  "NYC": "online",
  "LON": "offline"
}

Retrieve the status of a backup location with GET requests.

GET /v2/caches/{cacheName}/x-site/backups/{siteName}

Data Grid responds with the status of each node in the site in JSON format, as in the following example:

{
  "NodeA":"offline",
  "NodeB":"online"
}

Take backup locations offline with POST requests and the ?action=take-offline parameter.

POST /v2/caches/{cacheName}/x-site/backups/{siteName}?action=take-offline

Bring backup locations online with the ?action=bring-online parameter.

POST /v2/caches/{cacheName}/x-site/backups/{siteName}?action=bring-online

Push cache state to a backup location with the ?action=start-push-state parameter.

POST /v2/caches/{cacheName}/x-site/backups/{siteName}?action=start-push-state

Cancel state transfer operations with the ?action=cancel-push-state parameter.

POST /v2/caches/{cacheName}/x-site/backups/{siteName}?action=cancel-push-state

Retrieve status of state transfer operations with the ?action=push-state-status parameter.

GET /v2/caches/{cacheName}/x-site/backups?action=push-state-status

Data Grid responds with the status of state transfer for each backup location in JSON format, as in the following example:

{
   "NYC":"CANCELED",
   "LON":"OK"
}

Clear state transfer status for sending sites with the ?action=clear-push-state-status parameter.

POST /v2/caches/{cacheName}/x-site/local?action=clear-push-state-status

Sites go offline if certain conditions are met. Modify the take offline parameters to control when backup locations automatically go offline.

If the connection between two backup locations breaks, you can cancel state transfer on the site that is receiving the push.

Cancel state transfer from a remote site and keep the current state of the local cache with the ?action=cancel-receive-state parameter.

POST /v2/caches/{cacheName}/x-site/backups/{siteName}?action=cancel-receive-state

Retrieve the status of all backup locations from Cache Managers with GET requests.

GET /rest/v2/cache-managers/{cacheManagerName}/x-site/backups/

Data Grid responds with status in JSON format, as in the following example:

{
   "SFO-3":{
      "status":"online"
   },
   "NYC-2":{
      "status":"mixed",
      "online":[
         "CACHE_1"
      ],
      "offline":[
         "CACHE_2"
      ]
   }
}

Take backup locations offline with the ?action=take-offline parameter.

POST /rest/v2/cache-managers/{cacheManagerName}/x-site/backups/{siteName}?action=take-offline

Bring backup locations online with the ?action=bring-online parameter.

POST /rest/v2/cache-managers/{cacheManagerName}/x-site/backups/{siteName}?action=bring-online

Push state of all caches to remote sites with the ?action=start-push-state parameter.

POST /rest/v2/cache-managers/{cacheManagerName}/x-site/backups/{siteName}?action=start-push-state

Cancel ongoing state transfer operations with the ?action=cancel-push-state parameter.

POST /rest/v2/cache-managers/{cacheManagerName}/x-site/backups/{siteName}?action=cancel-push-state

Data Grid can register JMX MBeans that you can use to collect statistics and perform administrative operations. However, you must enable statistics separately to JMX otherwise Data Grid provides 0 values for all statistic attributes.

<cache-container>
  <jmx enabled="true" /> 1
</cache-container>

GlobalConfiguration globalConfig = new GlobalConfigurationBuilder()
  .jmx().enable() 1
  .build();

Perform cross-site operations via JMX clients.

See the Data Grid JMX Components documentation for details about available cross-site operations.

Collect Data Grid metrics with monitoring tools such as Prometheus.

static_configs:
    - targets: ['localhost:11222']

Data Grid provides JMX MBeans for cross-site replication that let you gather statistics and perform remote operations.

The org.infinispan:type=Cache component provides the following JMX MBeans:

The org.infinispan:type=CacheManager component includes the following JMX MBean:

For details about JMX MBeans along with descriptions of available operations and statistics, see the Data Grid JMX Components documentation.

Find user actions for log messages related to cross-site replication.