Chapter 11. Guaranteeing availability with anti-affinity

Kubernetes includes anti-affinity capabilities that protect workloads from single points of failure.

11.1. Anti-affinity strategies

Each Data Grid node in a cluster runs in a pod that runs on an OpenShift node in a cluster. Each Red Hat OpenShift node runs on a physical host system. Anti-affinity works by distributing Data Grid nodes across OpenShift nodes, ensuring that your Data Grid clusters remain available even if hardware failures occur.

Data Grid Operator offers two anti-affinity strategies:

kubernetes.io/hostname
Data Grid replica pods are scheduled on different OpenShift nodes.
topology.kubernetes.io/zone
Data Grid replica pods are scheduled across multiple zones.

Fault tolerance

Anti-affinity strategies guarantee cluster availability in different ways.

Note

The equations in the following section apply only if the number of OpenShift nodes or zones is greater than the number of Data Grid nodes.

Scheduling pods on different OpenShift nodes

Provides tolerance of x node failures for the following types of cache:

  • Replicated: x = spec.replicas - 1
  • Distributed: x = num_owners - 1

Scheduling pods across multiple zones

Provides tolerance of x zone failures when x zones exist for the following types of cache:

  • Replicated: x = spec.replicas - 1
  • Distributed: x = num_owners - 1
Note
spec.replicas
Defines the number of pods in each Data Grid cluster.
num_owners
Is the cache configuration attribute that defines the number of replicas for each entry in the cache.

11.2. Configuring anti-affinity

Specify where OpenShift schedules pods for your Data Grid clusters to ensure availability.

Procedure

  1. Add the spec.affinity block to your Infinispan CR.
  2. Configure anti-affinity strategies as necessary.
  3. Apply your Infinispan CR.

11.2.1. Anti-affinity strategy configurations

Configure anti-affinity strategies in your Infinispan CR to control where OpenShift schedules Data Grid replica pods.

Topology keysDescription

topologyKey: "topology.kubernetes.io/zone"

Schedules Data Grid replica pods across multiple zones.

topologyKey: "kubernetes.io/hostname"

Schedules Data Grid replica pods on different OpenShift nodes.

Schedule pods on different OpenShift nodes

The following is the anti-affinity strategy that Data Grid Operator uses if you do not configure the spec.affinity field in your Infinispan CR: 

spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app: infinispan-pod
              clusterName: <cluster_name>
              infinispan_cr: <cluster_name>
          topologyKey: "kubernetes.io/hostname"
Requiring different nodes

In the following example, OpenShift does not schedule Data Grid pods if different nodes are not available: 

spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: infinispan-pod
            clusterName: <cluster_name>
            infinispan_cr: <cluster_name>
        topologyKey: "topology.kubernetes.io/hostname"
Note

To ensure that you can schedule Data Grid replica pods on different OpenShift nodes, the number of OpenShift nodes available must be greater than the value of spec.replicas.

Schedule pods across multiple OpenShift zones

The following example prefers multiple zones when scheduling pods but schedules Data Grid replica pods on different OpenShift nodes if it is not possible to schedule across zones: 

spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app: infinispan-pod
              clusterName: <cluster_name>
              infinispan_cr: <cluster_name>
          topologyKey: "topology.kubernetes.io/zone"
      - weight: 90
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app: infinispan-pod
              clusterName: <cluster_name>
              infinispan_cr: <cluster_name>
          topologyKey: "kubernetes.io/hostname"
Requiring multiple zones

The following example uses the zone strategy only when scheduling Data Grid replica pods: 

spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: infinispan-pod
            clusterName: <cluster_name>
            infinispan_cr: <cluster_name>
        topologyKey: "topology.kubernetes.io/zone"