Site masters are the nodes in Data Grid clusters that are responsible for sending and receiving requests from backup locations.

If a node is not a site master, it must forward backup requests to a local site master. Only site masters can send requests to backup locations.

For optimal performance, you should configure all nodes as site masters. This increases the speed of backup requests because each node in the cluster can backup to remote sites directly without having to forward backup requests to site masters.

When Data Grid replicates data to backup locations, it waits until the operation completes before writing to the local cache.

You can control how Data Grid handles writes to the local cache if backup operations fail. For example, you can configure Data Grid to attempt to abort local writes and throw exceptions if backups to remote sites fail.

Synchronous backups also support two-phase commits with caches that participate in optimistic transactions. The first phase of the backup acquires a lock. The second phase commits the modification.

When Data Grid replicates data to backup locations, it does not wait until the operation completes before writing to the local cache.

Asynchronous backup operations and writes to the local cache are independent of each other. If backup operations fail, write operations to the local cache continue and no exceptions occur.

Synchronous backups offer the strongest guarantee of data consistency across sites. If strategy=sync, when cache.put() calls return you know the value is up to date in the local cache and in the backup locations.

The trade-off for this consistency is performance. Synchronous backups have much greater latency in comparison to asynchronous backups.

Asynchronous backups, on the other hand, do not add latency to client requests so they have no performance impact. However, if strategy=async, when cache.put() calls return you cannot be sure of the value in the backup locations is the same as in the local cache.

Conflicting entries can occur with Active/Active site configurations if clients write to the same entries at the same time but at different sites.

For example, client A writes to "k1" in LON at the same time that client B writes to "k1" in NYC. In this case, "k1" has a different value in LON than in NYC.

With synchronous replication, concurrent writes result in deadlocks because both sites lock the same key in different orders. To resolve deadlocks, client applications must wait until the locks time out.

With asynchronous replication, concurrent writes result in conflicting values because sites replicate after entries are modified locally. After replication occurs, there is no guarantee which value for "k1" exists at which site.

In all cases, inconsistencies in key values are resolved after the next non-conflicting put() operation updates the value.

Data Grid clusters use JGroups RELAY2 for inter-cluster discovery and communication.

<jgroups>
   <stack name="xsite" 1
          extends="udp"> 2
      <relay.RELAY2 xmlns="urn:org:jgroups" 3
                    site="LON" 4
                    max_site_masters="1000"/> 5
      <remote-sites default-stack="tcp"> 6
         <remote-site name="LON"/> 7
         <remote-site name="NYC"/>
      </remote-sites>
   </stack>
</jgroups>

<jgroups>
   <stack-file name="relay-global" path="jgroups-relay.xml"/> 1
   <stack name="xsite" extends="udp">
      <relay.RELAY2 site="LON" xmlns="urn:org:jgroups"
                    max_site_masters="10" 2
                    can_become_site_master="true"/>
      <remote-sites default-stack="relay-global">
         <remote-site name="LON"/>
         <remote-site name="NYC"/>
      </remote-sites>
   </stack>
</jgroups>

<config xmlns="urn:org:jgroups"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups-4.1.xsd">

    <!-- Use TCP for inter-cluster transport. -->
    <TCP bind_addr="127.0.0.1"
         bind_port="7200"
         port_range="30"

         thread_pool.min_threads="0"
         thread_pool.max_threads="8"
         thread_pool.keep_alive_time="5000"
    />

    <!-- Use TCPPING for inter-cluster discovery. -->
    <TCPPING timeout="3000"
             initial_hosts="127.0.0.1[7200]"
             port_range="3"
             ergonomics="false"/>

    <!-- Provide other configuration as required. -->
</config>

Take backup locations offline manually and bring them back online.

For more information and examples, run the help site command.

Transfer cache state to remote backup locations.

For more information and examples, run the help site command.

Retrieve the status of all backup locations with GET requests.

GET /v2/caches/{cacheName}/x-site/backups/

Data Grid responds with the status of each backup location in JSON format, as in the following example:

{
  "NYC": "online",
  "LON": "offline"
}

Retrieve the status of a backup location with GET requests.

GET /v2/caches/{cacheName}/x-site/backups/{siteName}

Data Grid responds with the status of each node in the site in JSON format, as in the following example:

{
  "NodeA":"offline",
  "NodeB":"online"
}

Take backup locations offline with GET requests and the ?action=take-offline parameter.

GET /v2/caches/{cacheName}/x-site/backups/{siteName}?action=take-offline

Bring backup locations online with the ?action=bring-online parameter.

GET /v2/caches/{cacheName}/x-site/backups/{siteName}?action=bring-online

Push cache state to a backup location with the ?action=start-push-state parameter.

GET /v2/caches/{cacheName}/x-site/backups/{siteName}?action=start-push-state

Cancel state transfer operations with the ?action=cancel-push-state parameter.

GET /v2/caches/{cacheName}/x-site/backups/{siteName}?action=cancel-push-state

Retrieve status of state transfer operations with the ?action=push-state-status parameter.

GET /v2/caches/{cacheName}/x-site/backups?action=push-state-status

Data Grid responds with the status of state transfer for each backup location in JSON format, as in the following example:

{
   "NYC":"CANCELED",
   "LON":"OK"
}

Clear state transfer status for sending sites with the ?action=clear-push-state-status parameter.

GET /v2/caches/{cacheName}/x-site/local?action=clear-push-state-status

Sites go offline if certain conditions are met. Modify the take offline parameters to control when backup locations automatically go offline.

If the connection between two backup locations breaks, you can cancel state transfer on the site that is receiving the push.

Cancel state transfer from a remote site and keep the current state of the local cache with the ?action=cancel-receive-state parameter.

GET /v2/caches/{cacheName}/x-site/backups/{siteName}?action=cancel-receive-state

Retrieve the status of all backup locations from Cache Managers with GET requests.

GET /rest/v2/cache-managers/{cacheManagerName}/x-site/backups/

Data Grid responds with status in JSON format, as in the following example:

{
   "SFO-3":{
      "status":"online"
   },
   "NYC-2":{
      "status":"mixed",
      "online":[
         "CACHE_1"
      ],
      "offline":[
         "CACHE_2"
      ]
   }
}

Take backup locations offline with the ?action=take-offline parameter.

GET /rest/v2/cache-managers/{cacheManagerName}/x-site/backups/{siteName}?action=take-offline

Bring backup locations online with the ?action=bring-online parameter.

GET /rest/v2/cache-managers/{cacheManagerName}/x-site/backups/{siteName}?action=bring-online

Push state of all caches to remote sites with the ?action=start-push-state parameter.

GET /rest/v2/cache-managers/{cacheManagerName}/x-site/backups/{siteName}?action=start-push-state

Cancel ongoing state transfer operations with the ?action=cancel-push-state parameter.

GET /rest/v2/cache-managers/{cacheManagerName}/x-site/backups/{siteName}?action=cancel-push-state