Chapter 3. Setting up Data Grid services

Use Data Grid Operator to create clusters of either Cache service or Data Grid service pods.

Important

If you do not specify a value for the spec.service.type field, Data Grid Operator creates Cache service pods by default.

You cannot change the spec.service.type field after you create pods. To change the service type, you must delete the existing pods and create new ones.

3.1. Service types

Services are stateful applications, based on the Data Grid Server image, that provide flexible and robust in-memory data storage.

3.1.1. Data Grid service

Deploy clusters of Data Grid service pods if you want to:

  • Back up data across global clusters with cross-site replication.
  • Create caches with any valid configuration.
  • Add file-based cache stores to save data in a persistent volume.
  • Query values across caches using the Data Grid Query API.
  • Use advanced Data Grid features and capabilities.

3.1.2. Cache service

Deploy clusters of Cache service pods if you want a low-latency data store with minimal configuration.

Cache service pods provide volatile storage only, which means you lose all data when you modify your Infinispan CR or update the version of your Data Grid cluster. However, if you only want to quickly provide applications with high-performance caching without the overhead of configuration then you can use Cache service pods to:

  • Automatically scale to meet capacity when data storage demands go up or down.
  • Synchronously distribute data to ensure consistency.
  • Replicates each entry in the cache across the cluster.
  • Store cache entries off-heap and use eviction for JVM efficiency.
  • Ensure data consistency with a default partition handling configuration.
Important

Red Hat recommends that you deploy Data Grid service pods instead of Cache service pods.

Cache service is planned for removal in the next version of the Infinispan CRD. Data Grid service remains under active development and will continue to benefit from new features and improved tooling to automate complex operations such as upgrading clusters and migrating data.

3.2. Creating Data Grid service pods

To use custom cache definitions along with Data Grid capabilities such as cross-site replication, create clusters of Data Grid service pods.

Procedure

  1. Create an Infinispan CR that sets spec.service.type: DataGrid and configures any other Data Grid service resources. 

    apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: example-infinispan
spec:
  replicas: 2
  service:
    type: DataGrid
  2. Apply your Infinispan CR to create the cluster.

3.2.1. Data Grid service CR

This topic describes the Infinispan CR for Data Grid service pods. 

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: example-infinispan
  annotations:
    infinispan.org/monitoring: 'true'
spec:
  replicas: 6
  service:
    type: DataGrid
    container:
      storage: 2Gi
      ephemeralStorage: false
      storageClassName: my-storage-class
    sites:
      local:
      name: azure
      expose:
        type: LoadBalancer
      locations:
      - name: azure
        url: openshift://api.azure.host:6443
        secretName: azure-token
      - name: aws
        clusterName: example-infinispan
        namespace: rhdg-namespace
        url: openshift://api.aws.host:6443
        secretName: aws-token
  security:
    endpointSecretName: endpoint-identities
    endpointEncryption:
        type: Secret
        certSecretName: tls-secret
  container:
    extraJvmOpts: "-XX:NativeMemoryTracking=summary"
    cpu: "1000m"
    memory: 1Gi
  logging:
    categories:
      org.infinispan: debug
      org.jgroups: debug
      org.jgroups.protocols.TCP: error
      org.jgroups.protocols.relay.RELAY2: error
  expose:
    type: LoadBalancer
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app: infinispan-pod
              clusterName: example-infinispan
              infinispan_cr: example-infinispan
          topologyKey: "kubernetes.io/hostname"
FieldDescription

metadata.name

Names your Data Grid cluster.

metadata.annotations.infinispan.org/monitoring

Automatically creates a ServiceMonitor for your cluster.

spec.replicas

Specifies the number of pods in your cluster.

spec.service.type

Configures the type Data Grid service. A value of DataGrid creates a cluster with Data Grid service pods.

spec.service.container

Configures the storage resources for Data Grid service pods.

spec.service.sites

Configures cross-site replication.

spec.security.endpointSecretName

Specifies an authentication secret that contains Data Grid user credentials.

spec.security.endpointEncryption

Specifies TLS certificates and keystores to encrypt client connections.

spec.container

Specifies JVM, CPU, and memory resources for Data Grid pods.

spec.logging

Configures Data Grid logging categories.

spec.expose

Controls how Data Grid endpoints are exposed on the network.

spec.affinity

Configures anti-affinity strategies that guarantee Data Grid availability.

3.3. Creating Cache service pods

Create Data Grid clusters with Cache service pods for a volatile, low-latency data store with minimal configuration.

Procedure

  1. Create an Infinispan CR that sets spec.service.type: Cache and configures any other Cache service resources. 

    apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: example-infinispan
spec:
  replicas: 2
  service:
    type: Cache
  2. Apply your Infinispan CR to create the cluster.

3.3.1. Cache service CR

This topic describes the Infinispan CR for Cache service pods. 

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: example-infinispan
  annotations:
    infinispan.org/monitoring: 'true'
spec:
  replicas: 2
  service:
    type: Cache
    replicationFactor: 2
  autoscale:
    maxMemUsagePercent: 70
    maxReplicas: 5
    minMemUsagePercent: 30
    minReplicas: 2
  security:
    endpointSecretName: endpoint-identities
    endpointEncryption:
        type: Secret
        certSecretName: tls-secret
  container:
    extraJvmOpts: "-XX:NativeMemoryTracking=summary"
    cpu: "2000m"
    memory: 1Gi
  logging:
    categories:
      org.infinispan: trace
      org.jgroups: trace
  expose:
    type: LoadBalancer
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app: infinispan-pod
              clusterName: example-infinispan
              infinispan_cr: example-infinispan
          topologyKey: "kubernetes.io/hostname"
FieldDescription

metadata.name

Names your Data Grid cluster.

metadata.annotations.infinispan.org/monitoring

Automatically creates a ServiceMonitor for your cluster.

spec.replicas

Specifies the number of pods in your cluster. If you enable autoscaling capabilities, this field specifies the initial number of pods.

spec.service.type

Configures the type Data Grid service. A value of Cache creates a cluster with Cache service pods.

spec.service.replicationFactor

Sets the number of copies for each entry across the cluster. The default for Cache service pods is two, which replicates each cache entry to avoid data loss.

spec.autoscale

Enables and configures automatic scaling.

spec.security.endpointSecretName

Specifies an authentication secret that contains Data Grid user credentials.

spec.security.endpointEncryption

Specifies TLS certificates and keystores to encrypt client connections.

spec.container

Specifies JVM, CPU, and memory resources for Data Grid pods.

spec.logging

Configures Data Grid logging categories.

spec.expose

Controls how Data Grid endpoints are exposed on the network.

spec.affinity

Configures anti-affinity strategies that guarantee Data Grid availability.

3.4. Automatic scaling

Data Grid Operator can monitor the default cache on Cache service pods to automatically scale clusters up or down, by creating or deleting pods based on memory usage.

Important

Automatic scaling is available for clusters of Cache service pods only. Data Grid Operator does not perform automatic scaling for clusters of Data Grid service pods.

When you enable automatic scaling, you define memory usage thresholds that let Data Grid Operator determine when it needs to create or delete pods. Data Grid Operator monitors statistics for the default cache and, when memory usage reaches the configured thresholds, scales your clusters up or down.

Maximum threshold

This threshold sets an upper boundary for the amount of memory that pods in your cluster can use before scaling up or performing eviction. When Data Grid Operator detects that any node reaches the maximum amount of memory that you configure, it creates a new node if possible. If Data Grid Operator cannot create a new node then it performs eviction when memory usage reaches 100 percent.

Minimum threshold

This threshold sets a lower boundary for memory usage across your Data Grid cluster. When Data Grid Operator detects that memory usage falls below the minimum, it shuts down pods.

Default cache only

Autoscaling capabilities work with the default cache only. If you plan to add other caches to your cluster, you should not include the autoscale field in your Infinispan CR. In this case you should use eviction to control the size of the data container on each node.

3.4.1. Configuring automatic scaling

If you create clusters with Cache service pods, you can configure Data Grid Operator to automatically scale clusters.

Procedure

  1. Add the spec.autoscale resource to your Infinispan CR to enable automatic scaling.

    Note

    Set a value of true for the autoscale.disabled field to disable automatic scaling.

  2. Configure thresholds for automatic scaling with the following fields:

    FieldDescription

    spec.autoscale.maxMemUsagePercent

    Specifies a maximum threshold, as a percentage, for memory usage on each node.

    spec.autoscale.maxReplicas

    Specifies the maximum number of Cache service pods for the cluster.

    spec.autoscale.minMemUsagePercent

    Specifies a minimum threshold, as a percentage, for cluster memory usage.

    spec.autoscale.minReplicas

    Specifies the minimum number of Cache service pods for the cluster.

    For example, add the following to your Infinispan CR: 

    spec:
  service:
    type: Cache
  autoscale:
    disabled: false
    maxMemUsagePercent: 70
    maxReplicas: 5
    minMemUsagePercent: 30
    minReplicas: 2
  3. Apply the changes.

3.5. Allocating storage resources

You can allocate storage for Data Grid service pods but not Cache service pods.

By default, Data Grid Operator allocates 1Gi for the persistent volume claim. However you should adjust the amount of storage available to Data Grid service pods so that Data Grid can preserve cluster state during shutdown.

Important

If available container storage is less than the amount of available memory, data loss can occur.

Procedure

  1. Allocate storage resources with the spec.service.container.storage field.

  2. Optionally configure the ephemeralStorage and storageClassName fields as required. 

    spec:
  service:
    type: DataGrid
    container:
      storage: 2Gi
      ephemeralStorage: false
      storageClassName: my-storage-class
  3. Apply the changes.
FieldDescription

spec.service.container.storage

Specifies the amount of storage for Data Grid service pods.

spec.service.container.ephemeralStorage

Defines whether storage is ephemeral or permanent. Set the value to true to use ephemeral storage, which means all data in storage is deleted when clusters shut down or restart. The default value is false, which means storage is permanent.

spec.service.container.storageClassName

Specifies the name of a StorageClass object to use for the persistent volume claim (PVC). If you include this field, you must specify an existing storage class as the value. If you do not include this field, the persistent volume claim uses the storage class that has the storageclass.kubernetes.io/is-default-class annotation set to true.

3.5.1. Persistent volume claims

Data Grid Operator creates a persistent volume claim (PVC) and mounts container storage at:
/opt/infinispan/server/data

Caches

When you create caches, Data Grid permanently stores their configuration so your caches are available after cluster restarts. This applies to both Cache service and Data Grid service pods.

Data

Data is always volatile in clusters of Cache service pods. When you shutdown the cluster, you permanently lose the data.

Use a file-based cache store, by adding the <file-store/> element to your Data Grid cache configuration, if you want Data Grid service pods to persist data during cluster shutdown.

3.6. JVM, CPU, and memory

You can set JVM options in Infinispan CR as well as CPU and memory allocation. 

spec:
  container:
    extraJvmOpts: "-XX:NativeMemoryTracking=summary"
    cpu: "1000m"
    memory: 1Gi
FieldDescription

spec.container.extraJvmOpts

Specifies JVM options.

spec.container.cpu

Allocates host CPU resources to Data Grid pods, measured in CPU units.

spec.container.memory

Allocates host memory to Data Grid pods, measured in bytes.

When Data Grid Operator creates Data Grid clusters, it uses spec.container.cpu and spec.container.memory to:

  • Ensure that OpenShift has sufficient capacity to run the Data Grid node. By default Data Grid Operator requests 512Mi of memory and 0.5 cpu from the OpenShift scheduler.
  • Constrain node resource usage. Data Grid Operator sets the values of cpu and memory as resource limits.

3.7. Adjusting log levels

Change levels for different Data Grid logging categories when you need to debug issues. You can also adjust log levels to reduce the number of messages for certain categories to minimize the use of container resources.

Procedure

  1. Configure Data Grid logging with the spec.logging.categories field in your Infinispan CR. 

    spec:
  logging:
    categories:
      org.infinispan: debug
      org.jgroups: debug
  2. Apply the changes.

  3. Retrieve logs from Data Grid pods as required. 

    $ oc logs -f $POD_NAME

3.7.1. Logging reference

Find information about log categories and levels.

Table 3.1. Log categories

Root categoryDescriptionDefault level

org.infinispan

Data Grid messages

info

org.jgroups

Cluster transport messages

info

Table 3.2. Log levels

Log levelDescription

trace

Provides detailed information about running state of applications. This is the most verbose log level.

debug

Indicates the progress of individual requests or activities.

info

Indicates overall progress of applications, including lifecycle events.

warn

Indicates circumstances that can lead to error or degrade performance.

error

Indicates error conditions that might prevent operations or activities from being successful but do not prevent applications from running.

Garbage collection (GC) messages

Data Grid Operator does not log GC messages by default. You can direct GC messages to stdout with the following JVM options: 

extraJvmOpts: "-Xlog:gc*:stdout:time,level,tags"

3.8. Adding labels to Data Grid resources

Attach key/value labels to pods and services that Data Grid Operator creates and manages. These labels help you identify relationships between objects to better organize and monitor Data Grid resources.

Note

Red Hat subscription labels are automatically applied to Data Grid pods.

Procedure

  1. Open your Infinispan CR for editing.
  2. Add any labels that you want Data Grid Operator to attach to resources with metadata.annotations.

  3. Add values for your labels with metadata.labels.

    1. Specify labels that you want to attach to services with the metadata.annotations.infinispan.org/targetLabels field.
    2. Specify labels that you want to attach to pods with the metadata.annotations.infinispan.org/podTargetLabels field.

    3. Define values for your labels with the metadata.labels fields. 

      apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  annotations:
    infinispan.org/targetLabels: svc-label1, svc-label2
    infinispan.org/podTargetLabels: pod-label1, pod-label2
  labels:
    svc-label1: svc-value1
    svc-label2: svc-value2
    pod-label1: pod-value1
    pod-label2: pod-value2
    # The operator does not attach these labels to resources.
    my-label: my-value
    environment: development
  4. Apply your Infinispan CR.

Additional resources