Chapter 10. Managing Arbitrated Replicated Volumes

10.1. Managing Arbiter Brick Size

A standard replica 3 volume has the same sized bricks in each set, however, an arbiter volume will have one brick in the brick set that can be smaller than the data bricks.

In order to better optimize the sizing of the Arbiter brick, Heketi allows the user to provide an average file size value that is used to calculate the final size of the Arbiter brick. This is done using the volume option “user.heketi.average-file-size NUM” where NUM is an integer value in KiB. By default Heketi uses a value of 64KiB.

To create an arbiter volume with a custom average file size using the heketi-cli command line tool the volume options "user.heketi.arbiter true" and "user.heketi.average-file-size 1024" must be provided.

For example:

# heketi-cli volume create --size=4 --gluster-volume-options='user.heketi.arbiter true,user.heketi.average-file-size 1024'

10.2. Managing Arbiter Brick Placement

To accomplish the task of controlling where arbiter bricks are placed, Heketi uses specific node and device tags. For the Arbiter feature, the tag "arbiter" can be applied to a node or device with the values of "supported", "required", or "disabled".

where:

  • supported: both arbiter bricks and data bricks are allowed.
  • required: only arbiter bricks are allowed, data bricks are rejected.
  • disabled: only data bricks are allowed, arbiter bricks are rejected.

Based on your use case, you can set tags on a node or a device.

For example, to use arbiter in order to split nodes such that arbiter nodes can act as dedicated "tiebreakers" between the nodes that host data, you can set a tag on the node.

The following example shows how to set tags on a device. The nodes have heterogeneous device types and you want to set a particular space saving pattern: one node with a small nvme device and two (or more) nodes with larger SSDs. To do this, set a tag on the device by identifying the small device as d1 (arbiter:required) and the larger devices as d2 and d3 (arbiter:disabled).

Note

A device without an explicit tag will automatically inherit the arbiter tag value from the node it is connected to. An explicit tag on the device always has priority over the node’s tag.

10.2.1. Setting Tags with the Heketi CLI

To set tags on nodes and device via the heketi-cli command line tool, execute the following commands:

Node

# heketi-cli node settags <node id> arbiter:<tag>

For example:

# heketi-cli node settags e2a792a43ca9a6bac4b9bfa792e89347 arbiter:disabled

Device

# heketi-cli device settags <device id> arbiter:<tag>

For example:

# heketi-cli device settags 167fe2831ad0a91f7173dac79172f8d7 arbiter:required

10.2.2. Removing Tags using Heketi CLI

If you want to remove the arbiter tags, then execute the following commands:

Node

# heketi-cli node rmtags <node id> arbiter

For example:

# heketi-cli node rmtags e2a792a43ca9a6bac4b9bfa792e89347 arbiter

Device

# heketi-cli device rmtags <device id> arbiter

For example:

# heketi-cli device rmtags 167fe2831ad0a91f7173dac79172f8d7 arbiter

10.2.3. Viewing Tags with the Heketi CLI

To view the tags, execute the following commands. If the node or device has any tags it will be displayed in a list below the heading "Tags":

Node

# heketi-cli node info <node id>

For example:

# heketi-cli node info e2a792a43ca9a6bac4b9bfa792e89347
Node Id: e2a792a43ca9a6bac4b9bfa792e89347
State: online
Cluster Id: ddb14817873c13c5bb42a5c04969daf9
Zone: 1
Management Hostname: 10.0.0.1
Storage Hostname: 10.0.0.1
Tags:
  arbiter: disabled
  test: demonstration
Devices:
Id:0b39f89c0677e8c0b796caf00204e726   Name:/dev/vdb            State:online    Size (GiB):500     Used (GiB):0       Free (GiB):500     Bricks:0
Id:167fe2831ad0a91f7173dac79172f8d7   Name:/dev/vdg            State:online    Size (GiB):500     Used (GiB):0       Free (GiB):500     Bricks:0

Device

# heketi-cli device info <device id>

For example:

# heketi-cli device info 167fe2831ad0a91f7173dac79172f8d7
Device Id: 167fe2831ad0a91f7173dac79172f8d7
Name: /dev/vdg
State: online
Size (GiB): 500
Used (GiB): 0
Free (GiB): 500
Tags:
  arbiter: required
  foobar: magic
Bricks:

10.3. Creating Persistent Volumes

For more information about creating persistent volumes, see https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html-single/operations_guide/#chap-Documentation-Red_Hat_Gluster_Storage_Container_Native_with_OpenShift_Platform-OpenShift_Creating_Persistent_Volumes-Dynamic_Prov

Important

In the Storage Class file ensure to add "user.heketi.arbiter true" under the volumeoptions parameter to create Arbiter volumes.

For example:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gluster-container
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://heketi-storage-project.cloudapps.mystorage.com"
  restuser: "admin"
  volumetype: "replicate:3"
  clusterid: "630372ccdc720a92c681fb928f27b53f,796e6db1981f369ea0340913eeea4c9a"
  secretNamespace: "default"
  secretName: "heketi-secret"
  volumeoptions: "user.heketi.arbiter true"
  volumenameprefix: "test-vol"
allowVolumeExpansion: "true"