Chapter 2. Data Grid Cross-Site Replication

Cross-site replication allows you to back up data from one Data Grid cluster to another. Learn the concepts to understand how Data Grid cross-site replication works before you configure your clusters.

2.1. Cross-Site Replication

Data Grid clusters running in different locations can discover and communicate with each other.

Sites are typically data centers in various geographic locations. Cross-site replication bridges Data Grid clusters in sites to form global clusters, as in the following diagram:

LON is a datacenter in London, England.
NYC is a datacenter in New York City, USA.

Note

Data Grid can form global clusters across two or more sites.

For example, configure a third Data Grid cluster running in San Francisco, SFO, as backup location for LON and NYC.

2.1.1. Site Masters

Site masters are the nodes in Data Grid clusters that are responsible for sending and receiving requests from backup locations.

If a node is not a site master, it must forward backup requests to a local site master. Only site masters can send requests to backup locations.

For optimal performance, you should configure all nodes as site masters. This increases the speed of backup requests because each node in the cluster can backup to remote sites directly without having to forward backup requests to site masters.

Note

Diagrams in this document illustrate Data Grid clusters with one site master because this is the default for the JGroups RELAY2 protocol. Likewise, a single site master is easier to illustrate because each site master in a cluster communicates with each site master in the remote cluster.

2.2. Adding Backups to Caches

Name remote sites as backup locations in your cache definitions.

For example, the following diagram shows three caches, "customers", "eu-orders", and "us-orders":

  • In LON, "customers" names NYC as a backup location.
  • In NYC, "customers" names LON as a backup location.
  • "eu-orders" and "us-orders" do not have backups and are local to the respective cluster.

2.3. Backup Strategies

Data Grid clusters can use different strategies for backing up data to remote sites.

Data Grid replicates across sites at the same time that writes to local caches occur. For example, if a client writes "k1" to LON, Data Grid backs up "k1" to NYC at the same time.

2.3.1. Synchronous Backups

When Data Grid replicates data to backup locations, it waits until the operation completes before writing to the local cache.

You can control how Data Grid handles writes to the local cache if backup operations fail. For example, you can configure Data Grid to attempt to abort local writes and throw exceptions if backups to remote sites fail.

Synchronous backups also support two-phase commits with caches that participate in optimistic transactions. The first phase of the backup acquires a lock. The second phase commits the modification.

Important

Two-phase commit with cross-site replication has a significant performance impact because it requires two round-trips across the network.

2.3.2. Asynchronous Backups

When Data Grid replicates data to backup locations, it does not wait until the operation completes before writing to the local cache.

Asynchronous backup operations and writes to the local cache are independent of each other. If backup operations fail, write operations to the local cache continue and no exceptions occur.

2.3.3. Synchronous vs Asynchronous Backups

Synchronous backups offer the strongest guarantee of data consistency across sites. If strategy=sync, when cache.put() calls return you know the value is up to date in the local cache and in the backup locations.

The trade-off for this consistency is performance. Synchronous backups have much greater latency in comparison to asynchronous backups.

Asynchronous backups, on the other hand, do not add latency to client requests so they have no performance impact. However, if strategy=async, when cache.put() calls return you cannot be sure of the value in the backup locations is the same as in the local cache.

2.4. Automatically Taking Backups Offline

You can configure backup locations to go offline automatically when the remote sites become unavailable. This prevents Data Grid nodes from continuously attempting to replicate data to offline backup locations, which results in error messages and consumes resources.

Timeout for backup operations

Backup configurations include timeout values for operations to replicate data. If operations do not complete before the timeout occurs, Infinispan records them as failures.

<backup site="NYC" strategy="ASYNC" timeout="10000"> 1
	...
</backup>
1
Operations to replicate data to NYC are recorded as failures if they do not complete after 10 seconds.

Number of failures

You can specify the number of consecutive failures that can occur before backup locations go offline.

For example, the following configuration for NYC sets five as the number of failed operations before it goes offline:

<backup site="NYC" strategy="ASYNC" timeout="10000">
	<take-offline after-failures="5"/> 1
</backup>
1
If a cluster attempts to replicate data to NYC and five consecutive operations fail, Data Grid automatically takes the backup offline.

Time to wait

You can also specify how long to wait before taking sites offline when backup operations fail. If a backup request succeeds before the wait time runs out, Data Grid does not take the site offline.

<backup site="NYC" strategy="ASYNC" timeout="10000">
	<take-offline after-failures="5"
                min-wait="15000"/> 1
</backup>
1
If a cluster attempts to replicate data to NYC and there are five consecutive failures and 15 seconds elapse after the first failed operation, Data Grid automatically takes the backup offline.
Tip

Set a negative or zero value for the after-failures attribute if you want to use only a minimum time to wait to take sites offline.

<take-offline after-failures="-1" min-wait="10000"/>

2.5. State Transfer

State transfer is an administrative operation that synchronizes data between sites.

For example, LON goes offline and NYC starts handling client requests. When you bring LON back online, the Data Grid cluster in LON does not have the same data as the cluster in NYC.

To ensure the data is consistent between LON and NYC, you can push state from NYC to LON.

  • State transfer is bidirectional. For example, you can push state from NYC to LON or from LON to NYC.
  • Pushing state to offline sites brings them back online.
  • State transfer overwrites only data that exists on both sites, the originating site and the receiving site. Data Grid does not delete data.

    For example, "k2" exists on LON and NYC. "k2" is removed from NYC while LON is offline. When you bring LON back online, "k2" still exists at that location. If you push state from NYC to LON, the transfer does not affect "k2" on LON.

    Tip

    To ensure contents of the cache are identical after state transfer, remove all data from the cache on the receiving site before pushing state. Use the clear() method.

  • State transfer does not overwrite updates to data that occur after you initiate the push.

    For example, "k1,v1" exists on LON and NYC. LON goes offline so you push state transfer to LON from NYC, which brings LON back online. Before state transfer completes, a client puts "k1,v2" on LON.

    In this case the state transfer from NYC does not overwrite "k1,v2" because that modification happened after you initiated the push.

Reference

2.6. Client Connections Across Sites

Clients can write to Data Grid clusters in either an Active/Passive or Active/Active configuration.

Active/Passive

The following diagram illustrates Active/Passive where Data Grid handles client requests from one site only:

In the preceding image:

  1. Client connects to the Data Grid cluster at LON.
  2. Client writes "k1" to the cache.
  3. The site master at LON, "n1", sends the request to replicate "k1" to the site master at NYC, "nA".

With Active/Passive, NYC provides data redundancy. If the Data Grid cluster at LON goes offline for any reason, clients can start sending requests to NYC. When you bring LON back online you can synchronize data with NYC and then switch clients back to LON.

Active/Active

The following diagram illustrates Active/Active where Data Grid handles client requests at two sites:

In the preceding image:

  1. Client A connects to the Data Grid cluster at LON.
  2. Client A writes "k1" to the cache.
  3. Client B connects to the Data Grid cluster at NYC.
  4. Client B writes "k2" to the cache.
  5. Site masters at LON and NYC send requests so that "k1" is replicated to NYC and "k2" is replicated to LON.

With Active/Active both NYC and LON replicate data to remote caches while handling client requests. If either NYC or LON go offline, clients can start sending requests to the online site. You can then bring offline sites back online, push state to synchronize data, and switch clients as required.

2.6.1. Concurrent Writes and Conflicting Entries

Conflicting entries can occur with Active/Active site configurations if clients write to the same entries at the same time but at different sites.

For example, client A writes to "k1" in LON at the same time that client B writes to "k1" in NYC. In this case, "k1" has a different value in LON than in NYC. After replication occurs, there is no guarantee which value for "k1" exists at which site.

To ensure data consistency, Data Grid uses a vector clock algorithm to detect conflicting entries during backup operations, as in the following illustration:

            LON       NYC

k1=(n/a)    0,0       0,0

k1=2        1,0  -->  1,0   k1=2

k1=3        1,1  <--  1,1   k1=3

k1=5        2,1       1,2   k1=8

                 -->  2,1 (conflict)
(conflict)  1,2  <--

Vector clocks are timestamp metadata that increment with each write to an entry. In the preceding example, 0,0 represents the initial value for the vector clock on "k1".

A client puts "k1=2" in LON and the vector clock is 1,0, which Data Grid replicates to NYC. A client then puts "k1=3" in NYC and the vector clock updates to 1,1, which Data Grid replicates to LON.

However if a client puts "k1=5" in LON at the same time that a client puts "k1=8" in NYC, Data Grid detects a conflicting entry because the vector value for "k1" is not strictly greater or less between LON and NYC.

When it finds conflicting entries, Data Grid uses the Java compareTo(String anotherString) method to compare site names. To determine which key takes priority, Data Grid selects the site name that is lexicographically less than the other. Keys from a site named AAA take priority over keys from a site named AAB and so on.

Following the same example, to resolve the conflict for "k1", Data Grid uses the value for "k1" that originates from LON. This results in "k1=5" in both LON and NYC after Data Grid resolves the conflict and replicates the value.

Tip

Prepend site names with numbers as a simple way to represent the order of priority for resolving conflicting entries; for example, 1LON and 2NYC.

2.7. Expiration and Cross-Site Replication

Data Grid expiration controls how long entries remain in the cache.

  • lifespan expiration is suitable for cross-site replication. When entries reach the maximum lifespan, Data Grid expires them independently of the remote sites.
  • max-idle expiration does not work with cross-site replication. Data Grid cannot determine when cache entries reach the idle timeout in remote sites.