Menu Close
Settings Close

Language and Page Formatting Options

Chapter 10. Setting up cross-site replication

Ensure service availability with Data Grid Operator by configuring cross-site replication to back up data between Data Grid clusters.

10.1. Using Data Grid Operator to manage cross-site connections

Data Grid Operator in one data center can discover a Data Grid cluster that Data Grid Operator manages in another data center. This discovery allows Data Grid to automatically form cross-site views and create global clusters.

The following illustration provides an example in which Data Grid Operator manages a Data Grid cluster at a data center in New York City, NYC. At another data center in London, LON, Data Grid Operator also manages a Data Grid cluster.

Data Grid Operator uses the Kubernetes API to establish a secure connection between the OpenShift Container Platform clusters in NYC and LON. Data Grid Operator then creates a cross-site replication service so Data Grid clusters can back up data across locations.

Important

Data Grid Operator in each OpenShift cluster must have network access to the remote Kubernetes API.

Note

When you configure automatic connections, Data Grid clusters do not start running until Data Grid Operator discovers all backup locations in the configuration.

Each Data Grid cluster has one site master node that coordinates all backup requests. Data Grid Operator identifies the site master node so that all traffic through the cross-site replication service goes to the site master.

If the current site master node goes offline then a new node becomes site master. Data Grid Operator automatically finds the new site master node and updates the cross-site replication service to forward backup requests to it.

10.1.1. Creating service account tokens

Generate service account tokens on each OpenShift cluster that acts as a backup location. Clusters use these tokens to authenticate with each other so Data Grid Operator can create a cross-site replication service.

Procedure

  1. Log in to an OpenShift cluster.
  2. Create a service account.

    For example, create a service account at LON:

    $ oc create sa lon
    serviceaccount/lon created
  3. Add the view role to the service account with the following command:

    $ oc policy add-role-to-user view system:serviceaccount:<namespace>:lon
  4. If you use a node port service to expose Data Grid clusters on the network, you must also add the cluster-reader role to the service account:

    $ oc adm policy add-cluster-role-to-user cluster-reader -z <service-account-name> -n <namespace>
  5. Repeat the preceding steps on your other OpenShift clusters.

10.1.2. Exchanging service account tokens

After you create service account tokens on your OpenShift clusters, you add them to secrets on each backup location. For example, at LON you add the service account token for NYC. At NYC you add the service account token for LON.

Prerequisites

  • Get tokens from each service account.

    Use the following command or get the token from the OpenShift Web Console:

    $ oc sa get-token lon
    
    eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...

Procedure

  1. Log in to an OpenShift cluster.
  2. Add the service account token for a backup location with the following command:

    $ oc create secret generic <token-name> --from-literal=token=<token>

    For example, log in to the OpenShift cluster at NYC and create a lon-token secret as follows:

    $ oc create secret generic lon-token --from-literal=token=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9...
  3. Repeat the preceding steps on your other OpenShift clusters.

10.1.3. Configuring Data Grid Operator to handle cross-site connections

Configure Data Grid Operator to establish cross-site views with Data Grid clusters.

Prerequisites

  • Create secrets that contain service account tokens for each backup location.

Procedure

  1. Create an Infinispan CR for each Data Grid cluster.
  2. Specify the name of the local site with spec.service.sites.local.name.
  3. Set the value of the spec.service.sites.local.expose.type field to either NodePort or LoadBalancer.
  4. Optionally configure ports with the following fields:

    • spec.service.sites.local.expose.nodePort if you use NodePort.
    • spec.service.sites.local.expose.port if you use LoadBalancer.
  5. Provide the name, URL, and secret for each Data Grid cluster that acts as a backup location with spec.service.sites.locations.
  6. If Data Grid cluster names or namespaces at the remote site do not match the local site, specify those values with the clusterName and namespace fields.

    The following are example Infinispan CR definitions for LON and NYC:

    • LON

      apiVersion: infinispan.org/v1
      kind: Infinispan
      metadata:
        name: example-infinispan
      spec:
        replicas: 3
        service:
          type: DataGrid
          sites:
            local:
              name: LON
              expose:
                type: LoadBalancer
                port: 65535
            locations:
              - name: NYC
                clusterName: <nyc_cluster_name>
                namespace: <nyc_cluster_namespace>
                url: openshift://api.rhdg-nyc.openshift-aws.myhost.com:6443
                secretName: nyc-token
        logging:
          categories:
            org.jgroups.protocols.TCP: error
            org.jgroups.protocols.relay.RELAY2: error
    • NYC

      apiVersion: infinispan.org/v1
      kind: Infinispan
      metadata:
        name: nyc-cluster
      spec:
        replicas: 2
        service:
          type: DataGrid
          sites:
            local:
              name: NYC
              expose:
                type: LoadBalancer
                port: 65535
            locations:
              - name: LON
                clusterName: example-infinispan
                namespace: rhdg-namespace
                url: openshift://api.rhdg-lon.openshift-aws.myhost.com:6443
                secretName: lon-token
        logging:
          categories:
            org.jgroups.protocols.TCP: error
            org.jgroups.protocols.relay.RELAY2: error
      Important

      Be sure to adjust logging categories in your Infinispan CR to decrease log levels for JGroups TCP and RELAY2 protocols. This prevents a large number of log files from uses container storage.

      spec:
        logging:
          categories:
            org.jgroups.protocols.TCP: error
            org.jgroups.protocols.relay.RELAY2: error
  7. Configure your Infinispan CRs with any other Data Grid service resources and then apply the changes.
  8. Verify that Data Grid clusters form a cross-site view.

    1. Retrieve the Infinispan CR.

      $ oc get infinispan -o yaml
    2. Check for the type: CrossSiteViewFormed condition.

Next steps

If your clusters have formed a cross-site view, you can start adding backup locations to caches.

10.1.4. Resources for managed cross-site connections

This topic describes resources for cross-site connections that Data Grid Operator manages.

spec:
  service:
    type: DataGrid
    sites:
      local:
        name: LON
        expose:
          type: LoadBalancer
      locations:
      - name: NYC
        clusterName: <nyc_cluster_name>
        namespace: <nyc_cluster_namespace>
        url: openshift://api.site-b.devcluster.openshift.com:6443
        secretName: nyc-token
FieldDescription

service.type: DataGrid

Data Grid supports cross-site replication with Data Grid service clusters only.

service.sites.local.name

Names the local site where a Data Grid cluster runs.

service.sites.local.expose.type

Specifies the network service for cross-site replication. Data Grid clusters use this service to communicate and perform backup operations. You can set the value to NodePort or LoadBalancer.

service.sites.local.expose.nodePort

Specifies a static port within the default range of 30000 to 32767 if you expose Data Grid through a NodePort service. If you do not specify a port, the platform selects an available one.

service.sites.local.expose.port

Specifies the network port for the service if you expose Data Grid through a LoadBalancer. The default port is 7900.

service.sites.locations

Provides connection information for all backup locations.

service.sites.locations.name

Specifies a backup location that matches .spec.service.sites.local.name.

service.sites.locations.url

Specifies the URL of the Kubernetes API for the backup location.

service.sites.locations.secretName

Specifies the secret that contains the service account token for the backup site.

service.sites.locations.clusterName

Specifies the cluster name at the backup location if it is different to the cluster name at the local site.

service.sites.locations.namespace

Specifies the namespace of the Data Grid cluster at the backup location if it does not match the namespace at the local site.

10.2. Manually connecting Data Grid clusters

You can specify static network connection details to perform cross-site replication with Data Grid clusters running outside OpenShift. Manual cross-site connections are necessary in any scenario where access to the Kubernetes API is not available outside the OpenShift cluster where Data Grid runs.

You can use both automatic and manual connections for Data Grid clusters in the same Infinispan CR. However, you must ensure that Data Grid clusters establish connections in the same way at each site.

Prerequisites

Manually connecting Data Grid clusters to form cross-site views requires predictable network locations for Data Grid services.

You need to know the network locations before they are created, which requires you to:

  • Have the host names and ports for each Data Grid cluster that you plan to configure as a backup location.
  • Have the host name of the <cluster-name>-site service for any remote Data Grid cluster that is running on OpenShift.
    You must use the <cluster-name>-site service to form a cross-site view between a cluster that Data Grid Operator manages and any other cluster.

Procedure

  1. Create an Infinispan CR for each Data Grid cluster.
  2. Specify the name of the local site with spec.service.sites.local.name.
  3. Set the value of the spec.service.sites.local.expose.type field to either NodePort or LoadBalancer.
  4. Optionally configure ports with the following fields:

    • spec.service.sites.local.expose.nodePort if you use NodePort.
    • spec.service.sites.local.expose.port if you use LoadBalancer.
  5. Provide the name and static URL for each Data Grid cluster that acts as a backup location with spec.service.sites.locations, for example:

    • LON

      apiVersion: infinispan.org/v1
      kind: Infinispan
      metadata:
        name: example-infinispan
      spec:
        replicas: 3
        service:
          type: DataGrid
          sites:
            local:
              name: LON
              expose:
                type: LoadBalancer
                port: 65535
            locations:
              - name: NYC
                url: infinispan+xsite://infinispan-nyc.myhost.com:7900
        logging:
          categories:
            org.jgroups.protocols.TCP: error
            org.jgroups.protocols.relay.RELAY2: error
    • NYC

      apiVersion: infinispan.org/v1
      kind: Infinispan
      metadata:
        name: example-infinispan
      spec:
        replicas: 2
        service:
          type: DataGrid
          sites:
            local:
              name: NYC
              expose:
                type: LoadBalancer
                port: 65535
            locations:
              - name: LON
                url: infinispan+xsite://infinispan-lon.myhost.com
        logging:
          categories:
            org.jgroups.protocols.TCP: error
            org.jgroups.protocols.relay.RELAY2: error
      Important

      Be sure to adjust logging categories in your Infinispan CR to decrease log levels for JGroups TCP and RELAY2 protocols. This prevents a large number of log files from uses container storage.

      spec:
        logging:
          categories:
            org.jgroups.protocols.TCP: error
            org.jgroups.protocols.relay.RELAY2: error
  6. Configure your Infinispan CRs with any other Data Grid service resources and then apply the changes.
  7. Verify that Data Grid clusters form a cross-site view.

    1. Retrieve the Infinispan CR.

      $ oc get infinispan -o yaml
    2. Check for the type: CrossSiteViewFormed condition.

Next steps

If your clusters have formed a cross-site view, you can start adding backup locations to caches.

10.2.1. Resources for manual cross-site connections

This topic describes resources for cross-site connections that you maintain manually.

spec:
  service:
    type: DataGrid
    sites:
      local:
        name: LON
        expose:
          type: LoadBalancer
          port: 65535
      locations:
      - name: NYC
        url: infinispan+xsite://infinispan-nyc.myhost.com:7900
FieldDescription

service.type: DataGrid

Data Grid supports cross-site replication with Data Grid service clusters only.

service.sites.local.name

Names the local site where a Data Grid cluster runs.

service.sites.local.expose.type

Specifies the network service for cross-site replication. Data Grid clusters use this service to communicate and perform backup operations. You can set the value to NodePort or LoadBalancer.

service.sites.local.expose.nodePort

Specifies a static port within the default range of 30000 to 32767 if you expose Data Grid through a NodePort service. If you do not specify a port, the platform selects an available one.

service.sites.local.expose.port

Specifies the network port for the service if you expose Data Grid through a LoadBalancer. The default port is 7900.

service.sites.locations

Provides connection information for all backup locations.

service.sites.locations.name

Specifies a backup location that matches .spec.service.sites.local.name.

service.sites.locations.url

Specifies the static URL for the backup location in the format of infinispan+xsite://<hostname>:<port>. The default port is 7900.

10.3. Configuring sites in the same OpenShift cluster

For evaluation and demonstration purposes, you can configure Data Grid to back up between nodes in the same OpenShift cluster.

Procedure

  1. Create an Infinispan CR for each Data Grid cluster.
  2. Specify the name of the local site with spec.service.sites.local.name.
  3. Set ClusterIP as the value of the spec.service.sites.local.expose.type field.
  4. Provide the name of the Data Grid cluster that acts as a backup location with spec.service.sites.locations.clusterName.
  5. If both Data Grid clusters have the same name, specify the namespace of the backup location with spec.service.sites.locations.namespace.

    apiVersion: infinispan.org/v1
    kind: Infinispan
    metadata:
      name: example-clustera
    spec:
      replicas: 1
      expose:
        type: LoadBalancer
      service:
        type: DataGrid
        sites:
          local:
            name: SiteA
            expose:
              type: ClusterIP
          locations:
            - name: SiteB
              clusterName: example-clusterb
              namespace: cluster-namespace
  6. Configure your Infinispan CRs with any other Data Grid service resources and then apply the changes.
  7. Verify that Data Grid clusters form a cross-site view.

    1. Retrieve the Infinispan CR.

      $ oc get infinispan -o yaml
    2. Check for the type: CrossSiteViewFormed condition.