Data Grid can back up data between clusters running in geographically dispersed data centers and across different cloud providers. Cross-site replication provides Data Grid with a global cluster view and:

Figure 1.1. Cross-site replication

Relay nodes are the nodes in Data Grid clusters that are responsible for sending and receiving requests from backup locations.

If a node is not a relay node, it must forward backup requests to a local relay node. Only relay nodes can send requests to backup locations.

For optimal performance, you should configure all nodes as relay nodes. This increases the speed of backup requests because each node in the cluster can backup to remote sites directly without having to forward backup requests to local relay nodes.

Data Grid caches include a backups configuration that let you name remote sites as backup locations.

For example, the following diagram shows three caches, "customers", "eu-orders", and "us-orders":

Data Grid replicates data between clusters at the same time that writes to caches occur. For example, if a client writes "k1" to LON, Data Grid backs up "k1" to NYC at the same time.

To back up data to a different cluster, Data Grid can use either a synchronous or asynchronous strategy.

Synchronous strategy

When Data Grid replicates data to backup locations, it writes to the cache on the local cluster and the cache on the remote cluster concurrently. With the synchronous strategy, Data Grid waits for both write operations to complete before returning.

You can control how Data Grid handles writes to the cache on the local cluster if backup operations fail. Data Grid can do the following:

Synchronous backups also support two-phase commits with caches that participate in optimistic transactions. The first phase of the backup acquires a lock. The second phase commits the modification.

Asynchronous strategy

When Data Grid replicates data to backup locations, it does not wait until the operation completes before writing to the local cache.

Asynchronous backup operations and writes to the local cache are independent of each other. If backup operations fail, write operations to the local cache continue and no exceptions occur. When this happens Data Grid also retries the write operation until the remote cluster disconnects from the cross-site view.

Synchronous vs asynchronous backups

Synchronous backups offer the strongest guarantee of data consistency across sites. If strategy=sync, when cache.put() calls return you know the value is up to date in the local cache and in the backup locations.

The trade-off for this consistency is performance. Synchronous backups have much greater latency in comparison to asynchronous backups.

Asynchronous backups, on the other hand, do not add latency to client requests so they have no performance impact. However, if strategy=async, when cache.put() calls return you cannot be sure of that the value in the backup location is the same as in the local cache.

Operations to replicate data across clusters are resource intensive, using excessive RAM and CPU. To avoid wasting resources Data Grid can take backup locations offline when they stop accepting requests after a specific period of time.

Data Grid takes remote sites offline based on the number of failed sequential requests and the time interval since the first failure. Requests are failed when the target cluster does not have any nodes in the cross-site view (JGroups bridge) or when a timeout expires before the target cluster acknowledges the request.

Backup timeouts

Backup configurations include timeout values for operations to replicate data between clusters. If operations do not complete before the timeout expires, Data Grid records them as failures.

In the following example, operations to replicate data to NYC are recorded as failures if they do not complete after 10 seconds:

<distributed-cache>
  <backups>
    <backup site="NYC"
            strategy="ASYNC"
            timeout="10000" />
  </backups>
</distributed-cache>

{
  "distributed-cache": {
    "backups": {
      "NYC" : {
        "backup" : {
          "strategy" : "ASYNC",
          "timeout" : "10000"
        }
      }
    }
  }
}

distributedCache:
  backups:
    NYC:
      backup:
        strategy: "ASYNC"
        timeout: "10000"

Number of failures

You can specify the number of consecutive failures that can occur before backup locations go offline.

In the following example, if a cluster attempts to replicate data to NYC and five consecutive operations fail, NYC automatically goes offline:

<distributed-cache>
  <backups>
    <backup site="NYC"
            strategy="ASYNC"
            timeout="10000">
      <take-offline after-failures="5"/>
    </backup>
  </backups>
</distributed-cache>

{
  "distributed-cache": {
    "backups": {
      "NYC" : {
        "backup" : {
          "strategy" : "ASYNC",
          "timeout" : "10000",
          "take-offline" : {
            "after-failures" : "5"
          }
        }
      }
    }
  }
}

distributedCache:
  backups:
    NYC:
      backup:
        strategy: "ASYNC"
        timeout: "10000"
        takeOffline:
          afterFailures: "5"

Time to wait

You can also specify how long to wait before taking sites offline when backup operations fail. If a backup request succeeds before the wait time runs out, Data Grid does not take the site offline.

One or two minutes is generally a suitable time to wait before automatically taking backup locations offline. If the wait period is too short then backup locations go offline too soon. You then need to bring clusters back online and perform state transfer operations to ensure data is in sync between the clusters.

A negative or zero value for the number of failures is equivalent to a value of 1. Data Grid uses only a minimum time to wait to take backup locations offline after a failure occurs, for example:

<take-offline after-failures="-1"
              min-wait="10000"/>

In the following example, if a cluster attempts to replicate data to NYC and there are more than five consecutive failures and 15 seconds elapse after the first failed operation, NYC automatically goes offline:

<distributed-cache>
  <backups>
    <backup site="NYC"
            strategy="ASYNC"
            timeout="10000">
      <take-offline after-failures="5" min-wait="15000"/>
    </backup>
  </backups>
</distributed-cache>

{
  "distributed-cache": {
    "backups": {
      "NYC" : {
        "backup" : {
          "strategy" : "ASYNC",
          "timeout" : "10000",
          "take-offline" : {
            "after-failures" : "5",
            "min-wait" : "15000"
          }
        }
      }
    }
  }
}

distributedCache:
  backups:
    NYC:
      backup:
        strategy: "ASYNC"
        timeout: "10000"
        takeOffline:
          afterFailures: "5"
          minWait: "15000"

State transfer is an administrative operation that synchronizes data between sites.

For example, LON goes offline and NYC starts handling client requests. When you bring LON back online, the Data Grid cluster in LON does not have the same data as the cluster in NYC.

To ensure the data is consistent between LON and NYC, you can push state from NYC to LON.

Automatic state transfer

By default you must manually perform cross-site state transfer operations with the CLI or via JMX or REST.

However, when using the asynchronous backup strategy, Data Grid can automatically perform cross-site state transfer operations. When it detects that a backup location has come back online, and the network connection is stable, Data Grid initiates bi-directional state transfer between backup locations. For example, Data Grid simultaneously transfers state from LON to NYC and NYC to LON.

Clients can write to Data Grid clusters in either an Active/Passive or Active/Active configuration.

Active/Passive

The following diagram illustrates Active/Passive where Data Grid handles client requests from one site only:

In the preceding image:

With Active/Passive, NYC provides data redundancy. If the Data Grid cluster at LON goes offline for any reason, clients can start sending requests to NYC. When you bring LON back online you can synchronize data with NYC and then switch clients back to LON.

Active/Active

The following diagram illustrates Active/Active where Data Grid handles client requests at two sites:

In the preceding image:

With Active/Active both NYC and LON replicate data to remote caches while handling client requests. If either NYC or LON go offline, clients can start sending requests to the online site. You can then bring offline sites back online, push state to synchronize data, and switch clients as required.

Backup strategies and client connections

If multiple clients attempt to write to the same entry concurrently, and the backup strategy is synchronous (strategy=sync), then deadlocks occur. However you can use the synchronous backup strategy with an Active/Passive configuration if both sites access different data sets, in which case there is no risk of deadlocks from concurrent writes.

            LON       NYC

k1=(n/a)    0,0       0,0

k1=2        1,0  -->  1,0   k1=2

k1=3        1,1  <--  1,1   k1=3

k1=5        2,1       1,2   k1=8

                 -->  2,1 (conflict)
(conflict)  1,2  <--

Expiration removes cache entries based on time. Data Grid provides two ways to configure expiration for entries:

Add JGroups RELAY2 to your transport layer so that Data Grid can replicate caches to backup locations.

JGroups RELAY2 stacks

The following configuration shows a JGroups RELAY2 stack that:

<infinispan>
  <jgroups>
    <stack name="xsite" extends="udp">
      <relay.RELAY2 xmlns="urn:org:jgroups"
                    site="LON"
                    max_site_masters="1000"/>
      <remote-sites default-stack="tcp">
        <remote-site name="LON"/>
        <remote-site name="NYC"/>
      </remote-sites>
    </stack>
  </jgroups>
  <cache-container>
    <transport cluster="${cluster.name}" stack="xsite"/>
  </cache-container>
</infinispan>

<infinispan>
  <jgroups>
    <stack name="relay-global" extends="tcp">
      <TCPPING initial_hosts="192.0.2.0[7800]"
               stack.combine="REPLACE"
               stack.position="MPING"/>
    </stack>
    <stack name="xsite" extends="udp">
      <relay.RELAY2 site="LON" xmlns="urn:org:jgroups"
                    max_site_masters="10"
                    can_become_site_master="true"/>
      <remote-sites default-stack="relay-global">
        <remote-site name="LON"/>
        <remote-site name="NYC"/>
      </remote-sites>
    </stack>
  </jgroups>
</infinispan>

Specify the names of remote sites so Data Grid can replicate data to caches on those clusters.

Backup configuration

The following example shows the "customers" cache configuration for the LON cluster:

<replicated-cache name="customers">
  <backups>
    <backup site="NYC"
            strategy="ASYNC" />
  </backups>
</replicated-cache>

{
  "replicated-cache": {
    "name": "customers",
    "backups": {
      "NYC": {
        "backup" : {
          "strategy" : "ASYNC"
        }
      }
    }
  }
}

replicatedCache:
  name: "customers"
  backups:
    NYC:
      backup:
        strategy: "ASYNC"

The following example shows the "customers" cache configuration for the NYC cluster:

<distributed-cache name="customers">
  <backups>
    <backup site="LON"
            strategy="ASYNC" />
  </backups>
</distributed-cache>

{
  "distributed-cache": {
    "name": "customers",
    "backups": {
      "LON": {
        "backup": {
          "strategy": "ASYNC"
        }
      }
    }
  }
}

distributedCache:
  name: "customers"
  backups:
    LON:
      backup:
        strategy: "ASYNC"

Data Grid replicates data between caches that have the same name by default. If you want Data Grid to replicate between caches with different names, you can explicitly declare the backup for each cache.

Backup for configuration

The following example configures the "eu-customers" cache to receive updates from the "customers" cache on the LON cluster:

<distributed-cache name="eu-customers">
  <backups>
    <backup site="LON"
            strategy="ASYNC" />
  </backups>
  <backup-for remote-cache="customers"
              remote-site="LON" />
</distributed-cache>

{
  "distributed-cache": {
    "name": "eu-customers",
    "backups": {
      "LON": {
        "backup": {
          "strategy": "ASYNC"
        }
      }
    },
    "backup-for" : {
      "remote-cache" : "customers",
      "remote-site" : "LON"
    }
  }
}

distributedCache:
  name: "eu-customers"
  backups:
    LON:
      backup:
        strategy: "ASYNC"
  backupFor:
    remoteCache: "customers"
    remoteSite: "LON"

Change cross-site state transfer settings to optimize performance and specify whether operations happen manually or automatically.

State transfer configuration

<distributed-cache name="eu-customers">
  <backups>
    <backup site="LON"
            strategy="ASYNC">
      <state-transfer chunk-size="600"
                      timeout="2400000"
                      max-retries="30"
                      wait-time="2000"
                      mode="AUTO"/>
    </backup>
  </backups>
</distributed-cache>

{
  "distributed-cache": {
    "name": "eu-customers",
    "backups": {
      "LON": {
        "backup": {
          "strategy": "ASYNC",
          "state-transfer": {
            "chunk-size": "600",
            "timeout": "2400000",
            "max-retries": "30",
            "wait-time": "2000",
            "mode": "AUTO"
          }
        }
      }
    }
  }
}

distributedCache:
  name: "eu-customers"
  backups:
    LON:
      backup:
        strategy: "ASYNC"
        stateTransfer:
          chunkSize: "600"
          timeout: "2400000"
          maxRetries: "30"
          waitTime: "2000"
          mode: "AUTO"

Configure Data Grid to use a different algorithm to resolve conflicting entries between backup locations.

Data Grid algorithms

The following example configuration uses the ALWAYS_REMOVE algorithm that deletes conflicting entries from both sites:

<distributed-cache>
   <backups merge-policy="ALWAYS_REMOVE">
      <backup site="LON" strategy="ASYNC"/>
   </backups>
</distributed-cache>

{
  "distributed-cache": {
    "backups": {
      "merge-policy": "ALWAYS_REMOVE",
      "LON": {
        "backup": {
          "strategy": "ASYNC"
        }
      }
    }
  }
}

distributedCache:
  backups:
    mergePolicy: "ALWAYS_REMOVE"
    LON:
      backup:
        strategy: "ASYNC"

Custom conflict resolution algorithms

If you create a custom XSiteEntryMergePolicy implementation, you can specify the fully qualified class name as the merge policy.

<distributed-cache>
   <backups merge-policy="org.mycompany.MyCustomXSiteEntryMergePolicy">
      <backup site="LON" strategy="ASYNC"/>
   </backups>
</distributed-cache>

{
  "distributed-cache": {
    "backups": {
      "merge-policy": "org.mycompany.MyCustomXSiteEntryMergePolicy",
      "LON": {
        "backup": {
          "strategy": "ASYNC"
        }
      }
    }
  }
}

distributedCache:
  backups:
    mergePolicy: "org.mycompany.MyCustomXSiteEntryMergePolicy"
    LON:
      backup:
        strategy: "ASYNC"

With the asynchronous backup strategy Data Grid stores metadata, known as tombstones, when it removes keys. Data Grid periodically runs a task to remove these tombstones and reduce excessive memory usage when backup locations no longer require the metadata. You can configure the frequency for this task by defining a target size for tombstone maps as well as the maximum delay between task runs.

Tombstone cleanup task configuration

<distributed-cache>
   <backups tombstone-map-size="512000" max-cleanup-delay="30000">
      <backup site="LON" strategy="ASYNC"/>
   </backups>
</distributed-cache>

{
  "distributed-cache": {
    "backups": {
      "tombstone-map-size": 512000,
      "max-cleanup-delay": 30000,
      "LON": {
        "backup": {
          "strategy": "ASYNC"
        }
      }
    }
  }
}

distributedCache:
  backups:
    tombstoneMapSize: 512000
    maxCleanupDelay: 30000
    LON:
      backup:
        strategy: "ASYNC"

When you set up Data Grid to perform cross-site replication, you should check log files to ensure that Data Grid clusters have successfully formed cross-site views.

For example, if a Data Grid cluster in LON has formed a cross-site view with a Data Grid cluster in NYC, logs include the following messages:

INFO  [org.infinispan.XSITE] (jgroups-5,<server-hostname>) ISPN000439: Received new x-site view: [NYC]
INFO  [org.infinispan.XSITE] (jgroups-7,<server-hostname>) ISPN000439: Received new x-site view: [LON, NYC]

Configure Hot Rod clients to use Data Grid clusters at different sites.

# Servers at the active site
infinispan.client.hotrod.server_list = LON_host1:11222,LON_host2:11222,LON_host3:11222

# Servers at the backup site
infinispan.client.hotrod.cluster.NYC = NYC_hostA:11222,NYC_hostB:11222,NYC_hostC:11222,NYC_hostD:11222

ConfigurationBuilder builder = new ConfigurationBuilder();
builder.addServers("LON_host1:11222;LON_host2:11222;LON_host3:11222")
       .addCluster("NYC")
       .addClusterNodes("NYC_hostA:11222;NYC_hostB:11222;NYC_hostC:11222;NYC_hostD:11222")

Take backup locations offline manually and bring them back online.

For more information and examples, run the help site command.

You can configure cross-site state transfer operations to happen automatically when Data Grid detects that backup locations come online. Alternatively you can use the default mode, which is to manually perform state transfer.

Transfer cache state to backup locations.

For more information and examples, run the help site command.

Retrieve the status of all backup locations with GET requests.

GET /rest/v2/caches/{cacheName}/x-site/backups/

Data Grid responds with the status of each backup location in JSON format, as in the following example:

{
  "NYC": {
    "status": "online"
  },
  "LON": {
    "status": "mixed",
    "online": [
      "NodeA"
    ],
    "offline": [
      "NodeB"
    ]
  }
}

Retrieve the status of a backup location with GET requests.

GET /rest/v2/caches/{cacheName}/x-site/backups/{siteName}

Data Grid responds with the status of each node in the site in JSON format, as in the following example:

{
  "NodeA":"offline",
  "NodeB":"online"
}

Take backup locations offline with POST requests and the ?action=take-offline parameter.

POST /rest/v2/caches/{cacheName}/x-site/backups/{siteName}?action=take-offline

Bring backup locations online with the ?action=bring-online parameter.

POST /rest/v2/caches/{cacheName}/x-site/backups/{siteName}?action=bring-online

Push cache state to a backup location with the ?action=start-push-state parameter.

POST /rest/v2/caches/{cacheName}/x-site/backups/{siteName}?action=start-push-state

Cancel state transfer operations with the ?action=cancel-push-state parameter.

POST /rest/v2/caches/{cacheName}/x-site/backups/{siteName}?action=cancel-push-state

Retrieve status of state transfer operations with the ?action=push-state-status parameter.

GET /rest/v2/caches/{cacheName}/x-site/backups?action=push-state-status

Data Grid responds with the status of state transfer for each backup location in JSON format, as in the following example:

{
   "NYC":"CANCELED",
   "LON":"OK"
}

Clear state transfer status for sending sites with the ?action=clear-push-state-status parameter.

POST /rest/v2/caches/{cacheName}/x-site/local?action=clear-push-state-status

Sites go offline if certain conditions are met. Modify the take offline parameters to control when backup locations automatically go offline.

If the connection between two backup locations breaks, you can cancel state transfer on the site that is receiving the push.

Cancel state transfer from a remote site and keep the current state of the local cache with the ?action=cancel-receive-state parameter.

POST /rest/v2/caches/{cacheName}/x-site/backups/{siteName}?action=cancel-receive-state

Retrieve the status of all backup locations from Cache Managers with GET requests.

GET /rest/v2/cache-managers/{cacheManagerName}/x-site/backups/

Data Grid responds with status in JSON format, as in the following example:

{
   "SFO-3":{
      "status":"online"
   },
   "NYC-2":{
      "status":"mixed",
      "online":[
         "CACHE_1"
      ],
      "offline":[
         "CACHE_2"
      ],
      "mixed": [
         "CACHE_3"
      ]
   }
}

GET /rest/v2/cache-managers/{cacheManagerName}/x-site/backups/{site}

Returns the status for a single backup location.

Take backup locations offline with the ?action=take-offline parameter.

POST /rest/v2/cache-managers/{cacheManagerName}/x-site/backups/{siteName}?action=take-offline

Bring backup locations online with the ?action=bring-online parameter.

POST /rest/v2/cache-managers/{cacheManagerName}/x-site/backups/{siteName}?action=bring-online

Check the state transfer mode with GET requests.

GET /rest/v2/caches/{cacheName}/x-site/backups/{site}/state-transfer-mode

Configure the state transfer mode with the ?action=set parameter.

POST /rest/v2/caches/{cacheName}/x-site/backups/{site}/state-transfer-mode?action=set&mode={mode}

Push state of all caches to remote sites with the ?action=start-push-state parameter.

POST /rest/v2/cache-managers/{cacheManagerName}/x-site/backups/{siteName}?action=start-push-state

Cancel ongoing state transfer operations with the ?action=cancel-push-state parameter.

POST /rest/v2/cache-managers/{cacheManagerName}/x-site/backups/{siteName}?action=cancel-push-state

Data Grid can register JMX MBeans that you can use to collect statistics and perform administrative operations. You must also enable statistics otherwise Data Grid provides 0 values for all statistic attributes in JMX MBeans.

JMX configuration

<infinispan>
  <cache-container statistics="true">
    <jmx enabled="true"
         domain="example.com"/>
  </cache-container>
</infinispan>

{
  "infinispan" : {
    "cache-container" : {
      "statistics" : "true",
      "jmx" : {
        "enabled" : "true",
        "domain" : "example.com"
      }
    }
  }
}

infinispan:
  cacheContainer:
    statistics: "true"
    jmx:
      enabled: "true"
      domain: "example.com"

Perform cross-site operations with JMX clients.

Data Grid provides JMX MBeans for cross-site replication that let you gather statistics and perform remote operations.

The org.infinispan:type=Cache component provides the following JMX MBeans:

The org.infinispan:type=CacheManager component includes the following JMX MBean:

For details about JMX MBeans along with descriptions of available operations and statistics, see the Data Grid JMX Components documentation.

Find user actions for log messages related to cross-site replication.