Chapter 3. Setting Up Data Grid Services

Use Data Grid Operator to create clusters of either Cache service or Data Grid service nodes.

3.1. Service Types

Services are stateful applications, based on the Data Grid server image, that provide flexible and robust in-memory data storage.

Cache service

Use Cache service if you want a volatile, low-latency data store with minimal configuration. Cache service nodes:

  • Automatically scale to meet capacity when data storage demands go up or down.
  • Synchronously distribute data to ensure consistency.
  • Replicates each entry in the cache across the cluster.
  • Store cache entries off-heap and use eviction for JVM efficiency.
  • Ensure data consistency with a default partition handling configuration.
Important

Because Cache service nodes are volatile you lose all data when you apply changes to the cluster with the Infinispan CR or update the Data Grid version.

Data Grid service

Use Data Grid service if you want to:

  • Back up data across global clusters with cross-site replication.
  • Create caches with any valid configuration.
  • Add file-based cache stores to save data in the persistent volume.
  • Use Data Grid search and other advanced capabilities.

3.2. Creating Cache service Nodes

By default, Data Grid Operator creates Data Grid clusters with Cache service nodes.

Procedure

  1. Create an Infinispan CR. 

    apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: example-infinispan
spec:
  replicas: 2
  service:
    type: Cache 1
    1
    Creates nodes Cache service nodes. This is the default for the Infinispan CR.
  2. Apply your Infinispan CR to create the cluster.

3.2.1. Configuring Automatic Scaling

If you create clusters with Cache service nodes, Data Grid Operator can automatically scale nodes up or down based on memory usage for the default cache.

Data Grid Operator monitors default caches on Cache service nodes. As you add data to the cache, memory usage increases. When it detects that the cluster needs additional capacity, Data Grid Operator creates new nodes rather than eviciting entries. Likewise, if it detects that memory usage is below a certain threshold, Data Grid Operator shuts down nodes.

Important

Automatic scaling works with the default cache only. If you plan to add other caches to your cluster, you should not include the autoscale field in your Infinispan CR. In this case you should use eviction to control the size of the data container on each node.

Procedure

  1. Add the spec.autoscale resource to your Infinispan CR to enable automatic scaling.

  2. Configure memory usage thresholds and number of nodes for your cluster with the autoscale field. 

    spec:
  ...
  service:
    type: Cache
  autoscale:
    maxMemUsagePercent: 70 1
    maxReplicas: 5 2
    minMemUsagePercent: 30 3
    minReplicas: 2 4
    1
    Configures the maximum threshold, as a percentage, for memory usage on each node. When Data Grid Operator detects that any node in the cluster reaches the threshold, it creates a new node if possible. If Data Grid Operator cannot create a new node then it performs eviction when memory usage reaches 100 percent.
    2
    Defines the maximum number of number of nodes for the cluster.
    3
    Configures the minimum threshold, as a percentage, for memory usage across the cluster. When Data Grid Operator detects that memory usage falls below the minimum, it shuts down nodes.
    4
    Defines the minimum number of number of nodes for the cluster.
  3. Apply the changes.

3.2.2. Configuring the Number of Owners

The number of owners controls how many copies of each cache entry are replicated across your Data Grid cluster. The default for Cache service nodes is two, which duplicates each entry to prevent data loss.

Procedure

  1. Specify the number of owners with the spec.service.replicationFactor resource in your Infinispan CR as follows: 

    spec:
  ...
  service:
    type: Cache
    replicationFactor: 3 1
    1
    Configures three replicas for each cache entry.
  2. Apply the changes.

3.2.3. Cache Service Resources

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  # Names the cluster.
  name: example-infinispan
spec:
  # Specifies the number of nodes in the cluster.
  replicas: 4
  service:
    # Configures the service type as Cache.
    type: Cache
    # Sets the number of replicas for each entry across the cluster.
    replicationFactor: 2
  # Enables and configures automatic scaling.
  autoscale:
    maxMemUsagePercent: 70
    maxReplicas: 5
    minMemUsagePercent: 30
    minReplicas: 2
  # Configures authentication and encryption.
  security:
    # Defines a secret with custom credentials.
    endpointSecretName: endpoint-identities
    # Adds a custom TLS certificate to encrypt client connections.
    endpointEncryption:
        type: Secret
        certSecretName: tls-secret
  # Sets container resources.
  container:
    extraJvmOpts: "-XX:NativeMemoryTracking=summary"
    cpu: "2000m"
    memory: 1Gi
  # Configures logging levels.
  logging:
    categories:
      org.infinispan: trace
      org.jgroups: trace
  # Configures how the cluster is exposed on the network.
  expose:
    type: LoadBalancer
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app: infinispan-pod
              clusterName: example-infinispan
              infinispan_cr: example-infinispan
          topologyKey: "kubernetes.io/hostname"

3.3. Creating Data Grid service Nodes

To use custom cache definitions along with Data Grid capabilities such as cross-site replication, create clusters of Data Grid service nodes.

Procedure

  1. Specify DataGrid as the value for spec.service.type in your Infinispan CR. 

    apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: example-infinispan
spec:
  replicas: 2
  service:
    type: DataGrid
    Note

    You cannot change the spec.service.type field after you create nodes. To change the service type, you must delete the existing nodes and create new ones.

  2. Configure nodes with any other Data Grid service resources.
  3. Apply your Infinispan CR to create the cluster.

3.3.1. Data Grid service Resources

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  # Names the cluster.
  name: example-infinispan
spec:
  # Specifies the number of nodes in the cluster.
  replicas: 6
  service:
    # Configures the service type as Data Grid.
    type: DataGrid
    # Configures storage resources.
    container:
      storage: 2Gi
      storageClassName: my-storage-class
    # Configures cross-site replication.
    sites:
      local:
      name: azure
      expose:
        type: LoadBalancer
      locations:
      - name: azure
        url: openshift://api.azure.host:6443
        secretName: azure-token
      - name: aws
        url: openshift://api.aws.host:6443
        secretName: aws-token
  # Configures authentication and encryption.
  security:
    # Defines a secret with custom credentials.
    endpointSecretName: endpoint-identities
    # Adds a custom TLS certificate to encrypt client connections.
    endpointEncryption:
        type: Secret
        certSecretName: tls-secret
  # Sets container resources.
  container:
    extraJvmOpts: "-XX:NativeMemoryTracking=summary"
    cpu: "1000m"
    memory: 1Gi
  # Configures logging levels.
  logging:
    categories:
      org.infinispan: debug
      org.jgroups: debug
      org.jgroups.protocols.TCP: error
      org.jgroups.protocols.relay.RELAY2: fatal
  # Configures how the cluster is exposed on the network.
  expose:
    type: LoadBalancer
  # Configures affinity and anti-affinity strategies.
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app: infinispan-pod
              clusterName: example-infinispan
              infinispan_cr: example-infinispan
          topologyKey: "kubernetes.io/hostname"

3.4. Adding Labels to Data Grid Resources

Attach key/value labels to pods and services that Data Grid Operator creates and manages. These labels help you identify relationships between objects to better organize and monitor Data Grid resources.

Note

Red Hat subscription labels are automatically applied to Data Grid pods.

Procedure

  1. Open your Infinispan CR for editing.
  2. Add any labels that you want Data Grid Operator to attach to resources with metadata.annotations.

  3. Add values for your labels with metadata.labels. 

    apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  annotations:
    # Add labels that you want to attach to services.
    infinispan.org/targetLabels: svc-label1, svc-label2
    # Add labels that you want to attach to pods.
    infinispan.org/podTargetLabels: pod-label1, pod-label2
  labels:
    # Add values for your labels.
    svc-label1: svc-value1
    svc-label2: svc-value2
    pod-label1: pod-value1
    pod-label2: pod-value2
    # The operator does not attach these labels to resources.
    my-label: my-value
    environment: development
  4. Apply your Infinispan CR.

Additional resources