Chapter 10. Managing Arbitrated Replicated Volumes
10.1. Managing Arbiter Brick Size
A standard replica 3 volume has the same sized bricks in each set, however, an arbiter volume will have one brick in the brick set that can be smaller than the data bricks.
In order to better optimize the sizing of the Arbiter brick, Heketi allows the user to provide an average file size value that is used to calculate the final size of the Arbiter brick. This is done using the volume option “user.heketi.average-file-size NUM” where NUM is an integer value in KiB. By default Heketi uses a value of 64KiB.
To create an arbiter volume with a custom average file size using the heketi-cli command line tool the volume options "user.heketi.arbiter true" and "user.heketi.average-file-size 1024" must be provided.
For example:
# heketi-cli volume create --size=4 --gluster-volume-options='user.heketi.arbiter true,user.heketi.average-file-size 1024'
10.2. Managing Arbiter Brick Placement
To accomplish the task of controlling where arbiter bricks are placed, Heketi uses specific node and device tags. For the Arbiter feature, the tag "arbiter" can be applied to a node or device with the values of "supported", "required", or "disabled".
where:
- supported: both arbiter bricks and data bricks are allowed.
- required: only arbiter bricks are allowed, data bricks are rejected.
- disabled: only data bricks are allowed, arbiter bricks are rejected.
Based on your use case, you can set tags on a node or a device.
For example, to use arbiter in order to split nodes such that arbiter nodes can act as dedicated "tiebreakers" between the nodes that host data, you can set a tag on the node.
The following example shows how to set tags on a device. The nodes have heterogeneous device types and you want to set a particular space saving pattern: one node with a small nvme device and two (or more) nodes with larger SSDs. To do this, set a tag on the device by identifying the small device as d1 (arbiter:required) and the larger devices as d2 and d3 (arbiter:disabled).
A device without an explicit tag will automatically inherit the arbiter tag value from the node it is connected to. An explicit tag on the device always has priority over the node’s tag.
10.2.1. Setting Tags with the Heketi CLI
To set tags on nodes and device via the heketi-cli command line tool, execute the following commands:
Node
# heketi-cli node settags <node id> arbiter:<tag>
For example:
# heketi-cli node settags e2a792a43ca9a6bac4b9bfa792e89347 arbiter:disabled
Device
# heketi-cli device settags <device id> arbiter:<tag>
For example:
# heketi-cli device settags 167fe2831ad0a91f7173dac79172f8d7 arbiter:required
10.2.2. Removing Tags using Heketi CLI
If you want to remove the arbiter tags, then execute the following commands:
Node
# heketi-cli node rmtags <node id> arbiter
For example:
# heketi-cli node rmtags e2a792a43ca9a6bac4b9bfa792e89347 arbiter
Device
# heketi-cli device rmtags <device id> arbiter
For example:
# heketi-cli device rmtags 167fe2831ad0a91f7173dac79172f8d7 arbiter
10.2.3. Viewing Tags with the Heketi CLI
To view the tags, execute the following commands. If the node or device has any tags it will be displayed in a list below the heading "Tags":
Node
# heketi-cli node info <node id>
For example:
# heketi-cli node info e2a792a43ca9a6bac4b9bfa792e89347 Node Id: e2a792a43ca9a6bac4b9bfa792e89347 State: online Cluster Id: ddb14817873c13c5bb42a5c04969daf9 Zone: 1 Management Hostname: 10.0.0.1 Storage Hostname: 10.0.0.1 Tags: arbiter: disabled test: demonstration Devices: Id:0b39f89c0677e8c0b796caf00204e726 Name:/dev/vdb State:online Size (GiB):500 Used (GiB):0 Free (GiB):500 Bricks:0 Id:167fe2831ad0a91f7173dac79172f8d7 Name:/dev/vdg State:online Size (GiB):500 Used (GiB):0 Free (GiB):500 Bricks:0
Device
# heketi-cli device info <device id>
For example:
# heketi-cli device info 167fe2831ad0a91f7173dac79172f8d7 Device Id: 167fe2831ad0a91f7173dac79172f8d7 Name: /dev/vdg State: online Size (GiB): 500 Used (GiB): 0 Free (GiB): 500 Tags: arbiter: required foobar: magic Bricks:
10.3. Creating Persistent Volumes
For more information about creating persistent volumes, see https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html-single/operations_guide/#chap-Documentation-Red_Hat_Gluster_Storage_Container_Native_with_OpenShift_Platform-OpenShift_Creating_Persistent_Volumes-Dynamic_Prov
In the Storage Class file ensure to add "user.heketi.arbiter true" under the volumeoptions parameter to create Arbiter volumes.
For example:
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gluster-container provisioner: kubernetes.io/glusterfs parameters: resturl: "http://heketi-storage-project.cloudapps.mystorage.com" restuser: "admin" volumetype: "replicate:3" clusterid: "630372ccdc720a92c681fb928f27b53f,796e6db1981f369ea0340913eeea4c9a" secretNamespace: "default" secretName: "heketi-secret" volumeoptions: "user.heketi.arbiter true" volumenameprefix: "test-vol" allowVolumeExpansion: "true"