Chapter 8. Configuring Cross-Site Replication

Set up global Data Grid clusters to back up data across sites.

8.1. Cross-Site Replication with Data Grid Operator

If you have Data Grid clusters running in separate locations, use Data Grid Operator to connect them so you can back up data across sites.

For example, in the following illustration, Data Grid Operator manages a Data Grid cluster at a data center in New York City, NYC. At another data center in London, LON, Data Grid Operator also manages a Data Grid cluster.

Data Grid Operator uses a Kubernetes API to establish a secure connection between the OpenShift Container Platform clusters in NYC and LON. Data Grid Operator then creates a cross-site replication service so Data Grid clusters can back up data across locations.

Each Data Grid cluster has one site master node that coordinates all backup requests. Data Grid Operator identifies the site master node so that all traffic through the cross-site replication service goes to the site master.

If the current site master node goes offline then a new node becomes site master. Data Grid Operator automatically finds the new site master node and updates the cross-site replication service to forward backup requests to it.

8.2. Creating Service Account Tokens

Generate service account tokens on each OpenShift cluster that acts as a backup location. Clusters use these tokens to authenticate with each other so Data Grid Operator can create a cross-site replication service.

Procedure

  1. Log in to an OpenShift cluster.

  2. Create a service account.

    For example, create a service account at LON: 

    $ oc create sa lon
serviceaccount/lon created

  3. Add the view role to the service account with the following command: 

    $ oc policy add-role-to-user view system:serviceaccount:<namespace>:lon
  4. Repeat the preceding steps on your other OpenShift clusters.

Additional resources

Using service accounts in applications

8.3. Exchanging Service Account Tokens

After you create service account tokens on your OpenShift clusters, you add them to secrets on each backup location. For example, at LON you add the service account token for NYC. At NYC you add the service account token for LON.

Prerequisites

  • Get tokens from each service account.

    Use the following command or get the token from the OpenShift Web Console: 

    $ oc sa get-token lon

eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...

Procedure

  1. Log in to an OpenShift cluster.

  2. Add the service account token for a backup location with the following command: 

    $ oc create secret generic <token-name> --from-literal=token=<token>

    For example, log in to the OpenShift cluster at NYC and create a lon-token secret as follows: 

    $ oc create secret generic lon-token --from-literal=token=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...
  3. Repeat the preceding steps on your other OpenShift clusters.

8.4. Configuring Data Grid Clusters for Cross-Site Replication

Configure Data Grid clusters as backup locations so that they can communicate over a dedicated JGroups transport channel for replicating data.

Prerequisites

  • Create secrets that contain service account tokens for each backup location.
  • Ensure that all clusters are Data Grid service nodes.

  • Ensure that OpenShift project names match.

    Important

    To perform cross-site replication, Data Grid Operator requires Data Grid clusters to have the same name and run in matching namespaces.

    For example, you create a cluster at LON in a project named xsite-cluster. The cluster at NYC must also run in a project named xsite-cluster.

Procedure

  1. Create an Infinispan CR for each Data Grid cluster.
  2. Specify a matching name for each Data Grid cluster with metadata.name.
  3. Specify the name of the local site with spec.service.sites.local.name.
  4. Set the expose service type for the local site with spec.service.sites.local.expose.type.

  5. Provide the name, URL, and secret for each Data Grid cluster that acts as a backup location with spec.service.sites.locations.

    The following are example Infinispan CR definitions for LON and NYC:

    • LON 

      apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: example-infinispan
spec:
  replicas: 3
  service:
    type: DataGrid
    sites:
      local:
        name: LON
        expose:
          type: LoadBalancer
      locations:
        - name: LON
          url: openshift://api.rhdg-lon.openshift-aws.myhost.com:6443
          secretName: lon-token
        - name: NYC
          url: openshift://api.rhdg-nyc.openshift-aws.myhost.com:6443
          secretName: nyc-token

    • NYC 

      apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: example-infinispan
spec:
  replicas: 2
  service:
    type: DataGrid
    sites:
      local:
        name: NYC
        expose:
          type: LoadBalancer
      locations:
        - name: NYC
          url: openshift://api.rhdg-nyc.openshift-aws.myhost.com:6443
          secretName: nyc-token
        - name: LON
          url: openshift://api.rhdg-lon.openshift-aws.myhost.com:6443
          secretName: lon-token

  6. Adjust logging levels for cross-site replication as follows: 

    ...
  logging:
    categories:
      org.jgroups.protocols.TCP: error
      org.jgroups.protocols.relay.RELAY2: fatal

    The preceding configuration decreases logging for JGroups TCP and RELAY2 protocols to reduce excessive messages about cluster backup operations, which can result in a large number of log files that use container storage.

  7. Configure nodes with any other Data Grid service resources.
  8. Apply the Infinispan CRs.

  9. Check node logs to verify that Data Grid clusters form a cross-site view, for example: 

    $ oc logs example-infinispan-0 | grep x-site

INFO  [org.infinispan.XSITE] (jgroups-5,example-infinispan-0-<id>) ISPN000439: Received new x-site view: [NYC]
INFO  [org.infinispan.XSITE] (jgroups-7,example-infinispan-0-<id>) ISPN000439: Received new x-site view: [NYC, LON]

Next steps

If your clusters have formed a cross-site view, you can start adding backup locations to caches.

Additional resources

8.4.1. Cross-Site Replication Resources

spec:
  ...
  service:
    type: DataGrid 1
    sites:
      local:
        name: LON 2
        expose:
          type: LoadBalancer 3
      locations: 4
      - name: LON 5
        url: openshift://api.site-a.devcluster.openshift.com:6443 6
        secretName: lon-token 7
      - name: NYC
        url: openshift://api.site-b.devcluster.openshift.com:6443
        secretName: nyc-token
  logging:
    categories:
      org.jgroups.protocols.TCP: error 8
      org.jgroups.protocols.relay.RELAY2: fatal 9
1
Specifies Data Grid service. Data Grid supports cross-site replication with Data Grid service clusters only.
2
Names the local site for a Data Grid cluster.
3
Specifies LoadBalancer as the service that handles communication between backup locations.
4
Provides connection information for all backup locations.
5
Specifies a backup location that matches spec.service.sites.local.name.
6
Specifies the URL of the OpenShift API for the backup location.
7
Specifies the secret that contains the service account token for the backup site.
8
Logs error messages for the JGroups TCP protocol.
9
Logs fatal messages for the JGroups RELAY2 protocol.