Language:
Format:

Chapter 6. Upgrading your Red Hat Openshift Container Storage in Converged Mode

This chapter describes the procedure to upgrade your environment from Container Storage in Converged Mode 3.10 to Red Hat Openshift Container Storage in Converged Mode 3.11.

Note

New registry name registry.redhat.io is used throughout in this Guide. However, if you have not migrated to the new registry yet then replace all occurrences of registry.redhat.io with registry.access.redhat.com where ever applicable.
Follow the same upgrade procedure to upgrade your environment from Red Hat Openshift Container Storage in Converged Mode 3.11.0 and above to Red Hat Openshift Container Storage in Converged Mode 3.11.8. Ensure that the correct image and version numbers are configured before you start the upgrade process.
The valid images for Red Hat Openshift Container Storage 3.11.8 are:
- registry.redhat.io/rhgs3/rhgs-server-rhel7:v3.11.8
- registry.redhat.io/rhgs3/rhgs-volmanager-rhel7:v3.11.8
- registry.redhat.io/rhgs3/rhgs-gluster-block-prov-rhel7:v3.11.8
- registry.redhat.io/rhgs3/rhgs-s3-server-rhel7:v3.11.8

6.1. Upgrading the pods in the glusterfs group

The following sections provide steps to upgrade your Glusterfs pods.

6.1.1. Prerequisites

Ensure the following prerequisites are met:

Section 3.1.3, “Red Hat OpenShift Container Platform and Red Hat Openshift Container Storage Requirements”
Ensure to have the supported versions of OpenShift Container Platform with Red Hat Gluster Storage Server and Red Hat Openshift Container Storage. For more information on supported versions, see Section 3.1.1, “Supported Versions”
Ensure to run the following command to get the latest versions of Ansible templates.
```
# yum update openshift-ansible
```

Note

For deployments using cns-deploy tool, the templates are available in the following location:

gluster template - /usr/share/heketi/templates/glusterfs-template.yaml
heketi template - /usr/share/heketi/templates/heketi-template.yaml
glusterblock-provisioner template - /usr/share/heketi/templates/glusterblock-provisioner.yaml

For deployments using ansible playbook the templates are available in the following location:

gluster template - /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/glusterfs-template.yml
heketi template - /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/heketi-template.yml
glusterblock-provisioner template - /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/glusterblock-provisioner.yml

6.1.2. Restoring original label values for /dev/log

Note

Follow this procedure only if you are upgrading your environment from Red Hat Container Native Storage 3.9 to Red Hat Openshift Container Storage 3.11.8.

Skip this procedure if you are upgrading your environment from Red Hat Openshift Container Storage 3.10 and above to Red Hat Openshift Container Storage 3.11.8.

To restore the original selinux label, execute the following commands:

Create a directory and soft links on all nodes that run gluster pods:

# mkdir /srv/<directory_name>
# cd /srv/<directory_name>/   # same dir as above
# ln -sf /dev/null systemd-tmpfiles-setup-dev.service
# ln -sf /dev/null systemd-journald.service
# ln -sf /dev/null systemd-journald.socket

Edit the daemonset that creates the glusterfs pods on the node which has oc client:

# oc edit daemonset <daemonset_name>

Under volumeMounts section add a mapping for the volume:

- mountPath: /usr/lib/systemd/system/systemd-journald.service
  name: systemd-journald-service
- mountPath: /usr/lib/systemd/system/systemd-journald.socket
  name: systemd-journald-socket
- mountPath: /usr/lib/systemd/system/systemd-tmpfiles-setup-dev.service
name: systemd-tmpfiles-setup-dev-service

Under volumes section add a new host path for each service listed:

Note

The path mentioned in here should be the same as mentioned in Step 1.

- hostPath:
   path: /srv/<directory_name>/systemd-journald.socket
   type: ""
  name: systemd-journald-socket
- hostPath:
   path: /srv/<directory_name>/systemd-journald.service
   type: ""
  name: systemd-journald-service
- hostPath:
   path: /srv/<directory_name>/systemd-tmpfiles-setup-dev.service
   type: ""
name: systemd-tmpfiles-setup-dev-service

Run the following command on all nodes that run gluster pods. This will reset the label:
```
# restorecon /dev/log
```

Execute the following command to check the status of self heal for all volumes:

# oc rsh <gluster_pod_name>
# for each_volume in `gluster volume list`; do gluster volume heal $each_volume info ; done  | grep  "Number of entries: [^0]$"

Wait for self-heal to complete.

Execute the following command and ensure that the bricks are not more than 90% full:
```
# df -kh | grep -v ^Filesystem | awk '{if(int($5)>90) print $0}'
```
Note
If the bricks are close to 100% utilization, then the Logical Volume Manager(LVM) activation for these bricks may take a long time or can get stuck once the pod or node is rebooted. It is advised to bring down the utilization of that brick or expand the physical volume(PV) that is using the logical volume(LV).
Note
The df command is not applicable to bricks that belong to Block Hosting Volume(BHV). On a BHV , the used size of the bricks generated by the df command is the added size of the blockvolumes of that Gluster volume, it is not the size of the data that resides in the blockvolumes. For more information refer to How To Identify Block Volumes and Block Hosting Volumes in Openshift Container Storage.
Execute the following command on any one of the gluster pods to set the maximum number of bricks (250) that can run on a single instance of glusterfsd process:
```
# gluster volume set all cluster.max-bricks-per-process 250
```
1. Execute the following command on any one of the gluster pods to ensure that the option is set correctly:
```
# gluster volume get all cluster.max-bricks-per-process
```
  For example:
```
# gluster volume get all cluster.max-bricks-per-process
cluster.max-bricks-per-process 250
```
Execute the following command on the node which has oc client to delete the gluster pod:
```
# oc delete pod <gluster_pod_name>
```
To verify if the pod is ready, execute the following command:
```
# oc get pods -l glusterfs=storage-pod
```
Login to the node hosting the pod and check the selinux label of /dev/log
```
# ls -lZ /dev/log
```
The output should show devlog_t label
For example:
```
#  ls -lZ /dev/log
srw-rw-rw-. root root system_u:object_r:devlog_t:s0    /dev/log
```
Exit the node.

In the gluster pod, check if the label value is devlog_t:

# oc rsh <gluster_pod_name>
# ls -lZ /dev/log

For example:

#  ls -lZ /dev/log
srw-rw-rw-. root root system_u:object_r:devlog_t:s0    /dev/log

Perform steps 4 to 9 for other pods.

6.1.3. Upgrading if existing version deployed by using cns-deploy

6.1.3.1. Upgrading cns-deploy and Heketi Server

The following commands must be executed on the client machine.

Execute the following command to update the heketi client and cns-deploy packages:
```
# yum update cns-deploy -y
# yum update heketi-client -y
```
Backup the Heketi database file
```
# heketi-cli db dump > heketi-db-dump-$(date -I).json
```
- Execute the following command to get the current HEKETI_ADMIN_KEY.
  The OCS admin can choose to set any phrase for user key as long as it is not used by their infrastructure. It is not used by any of the OCS default installed resources.
```
oc get secret <heketi-admin-secret> -o jsonpath='{.data.key}'|base64 -d;echo
```
Execute the following command to delete the heketi template.
```
# oc delete templates heketi
```

Execute the following command to install the heketi template.

oc create -f /usr/share/heketi/templates/heketi-template.yaml
template "heketi" created

Execute the following command to grant the heketi Service Account the necessary privileges.

# oc policy add-role-to-user edit system:serviceaccount:<project_name>:heketi-service-account
# oc adm policy add-scc-to-user privileged -z heketi-service-account

For example,

# oc policy add-role-to-user edit system:serviceaccount:storage-project:heketi-service-account
# oc adm policy add-scc-to-user privileged -z heketi-service-account

Execute the following command to generate a new heketi configuration file.
```
# sed -e "s/\${HEKETI_EXECUTOR}/kubernetes/" -e "s#\${HEKETI_FSTAB}#/var/lib/heketi/fstab#" -e "s/\${SSH_PORT}/22/" -e "s/\${SSH_USER}/root/" -e "s/\${SSH_SUDO}/false/" -e "s/\${BLOCK_HOST_CREATE}/true/" -e "s/\${BLOCK_HOST_SIZE}/500/" "/usr/share/heketi/templates/heketi.json.template" > heketi.json
```
- The BLOCK_HOST_SIZE parameter controls the size (in GB) of the automatically created Red Hat Gluster Storage volumes hosting the gluster-block volumes (For more information, see https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html-single/operations_guide/index#Block_Storage). This default configuration will dynamically create block-hosting volumes of 500GB in size as more space is required.
- Alternatively, copy the file /usr/share/heketi/templates/heketi.json.template to heketi.json in the current directory and edit the new file directly, replacing each "${VARIABLE}" string with the required parameter.
  Note
  JSON formatting is strictly required (e.g. no trailing spaces, booleans in all lowercase).
Execute the following command to create a secret to hold the configuration file.
```
# oc create secret generic <heketi-config-secret> --from-file=heketi.json
```
Note
If the heketi-config-secret file already exists, then delete the file and run the following command.
Execute the following command to delete the deployment configuration, service, and route for heketi:
```
# oc delete deploymentconfig,service,route heketi
```
Note
The names of these parameters can be referenced from output of the following command:
```
# oc get all | grep heketi
```

Edit the heketi template.

Edit the HEKETI_USER_KEY and HEKETI_ADMIN_KEY parameters.

# oc edit template heketi
parameters:
- description: Set secret for those creating volumes as type user
  displayName: Heketi User Secret
  name: HEKETI_USER_KEY
  value: <heketiuserkey>
- description: Set secret for administration of the Heketi service as user admin
  displayName: Heketi Administrator Secret
  name: HEKETI_ADMIN_KEY
  value: <adminkey>
- description: Set the executor type, kubernetes or ssh
  displayName: heketi executor type
  name: HEKETI_EXECUTOR
  value: kubernetes
- description: Set the hostname for the route URL
  displayName: heketi route name
  name: HEKETI_ROUTE
  value: heketi-storage
- displayName: heketi container image name
  name: IMAGE_NAME
  required: true
  value: registry.redhat.io/rhgs3/rhgs-volmanager-rhel7
- displayName: heketi container image version
  name: IMAGE_VERSION
  required: true
  value: v3.11.8
- description: A unique name to identify this heketi service, useful for running
    multiple heketi instances
  displayName: GlusterFS cluster name
  name: CLUSTER_NAME
  value: storage

Note

If a cluster has more than 1000 volumes refer to How to change the default PVS limit in Openshift Container Storage and add the required parameters before proceeding with the upgrade.

Add an ENV with the name HEKETI_LVM_WRAPPER and value /usr/sbin/exec-on-host.

- description: Heketi can use a wrapper to execute LVM commands, i.e. run commands
in the host namespace instead of in the Gluster container.
displayName: Wrapper for executing LVM commands
name: HEKETI_LVM_WRAPPER
value: /usr/sbin/exec-on-host

Add an ENV with the name HEKETI_DEBUG_UMOUNT_FAILURES and value true.

- description: When unmounting a brick fails, Heketi will not be able to cleanup the
Gluster volume completely. The main causes for preventing to unmount a brick,
seem to originate from Gluster processes. By enabling this option, the heketi.log
will contain the output of 'lsof' to aid with debugging of the Gluster processes
and help with identifying any files that may be left open.
displayName: Capture more details in case brick unmounting fails
name: HEKETI_DEBUG_UMOUNT_FAILURES
required=true

Add an ENV with the name HEKETI_CLI_USER and value admin.
Add an ENV with the name HEKETI_CLI_KEY and the same value provided for the ENV HEKETI_ADMIN_KEY.

Replace the value under IMAGE_VERSION with v3.11.5 or v3.11.8 depending on the version you want to upgrade to.

- displayName: heketi container image name
  name: IMAGE_NAME
  required: true
  value: registry.redhat.io/rhgs3/rhgs-volmanager-rhel7
- displayName: heketi container image version
  name: IMAGE_VERSION
  required: true
  value: v3.11.8

Execute the following command to deploy the Heketi service, route, and deployment configuration which will be used to create persistent volumes for OpenShift:
```
# oc process heketi | oc create -f -

  service "heketi" created
  route "heketi" created
deploymentconfig "heketi" created
```
Note
It is recommended that the heketidbstorage volume be tuned for db workloads. Newly installed Openshift Container Storage deployments tune the heketidbstorage volume automatically. For older deployments, follow the KCS article Planning to run containerized DB or nosql workloads on Openshift Container Storage? and perform the volume set operation for the volume heketidbstorage.

Execute the following command to verify that the containers are running:

# oc get pods

For example:

# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
heketi-storage-4-9fnvz                        2/2       Running   0          8d

6.1.3.2. Upgrading the Red Hat Gluster Storage Pods

The following commands must be executed on the client machine.

Following are the steps for updating a DaemonSet for glusterfs:

Execute the following steps to stop the Heketi pod to prevent it from accepting any new request for volume creation or volume deletion:
1. Execute the following command to access your project:
```
# oc project <project_name>
```
  For example:
```
# oc project storage-project
```
2. Execute the following command to get the DeploymentConfig:
```
# oc get ds
```
3. Execute the following command to set heketi server to accept requests only from the local-client:
```
# heketi-cli server mode set local-client
```
4. Wait for the ongoing operations to complete and execute the following command to monitor if there are any ongoing operations:
```
# heketi-cli server operations info
```
5. Execute the following command to reduce the replica count from 1 to 0. This brings down the Heketi pod:
```
# oc scale dc <heketi_dc> --replicas=0
```
6. Execute the following command to verify that the heketi pod is no longer present:
```
# oc get pods
```
Execute the following command to find the DaemonSet name for gluster
```
# oc get ds
```
Execute the following command to delete the DaemonSet:
```
# oc delete ds <ds-name> --cascade=false
```
Using --cascade=false option while deleting the old DaemonSet does not delete the gluster pods but deletes only the DaemonSet. After deleting the old DaemonSet, you must load the new one. When you manually delete the old pods, the new pods which are created will have the configurations of the new DaemonSet.
For example,
```
# oc delete ds glusterfs --cascade=false
daemonset "glusterfs" deleted
```

Execute the following commands to verify all the old pods are up:

# oc get pods

For example,

# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d

Execute the following command to delete the old glusterfs template.

# oc delete templates glusterfs

For example,

# oc delete templates glusterfs
template “glusterfs” deleted

Execute the following command to register new glusterfs template.

# oc create -f /usr/share/heketi/templates/glusterfs-template.yaml

For example,

# oc create -f /usr/share/heketi/templates/glusterfs-template.yaml
  template “glusterfs” created

Label all the OpenShift Container Platform nodes that has the Red Hat Gluster Storage pods:
1. Check if the nodes are labelled with the appropriate label by using the following command:
```
# oc get nodes -l glusterfs=storage-host
```

Edit the glusterfs template.

Execute the following command:
```
# oc edit template glusterfs
```

Add the following lines under volume mounts:

 - name: kernel-modules
   mountPath: "/usr/lib/modules"
   readOnly: true
 - name: host-rootfs
   mountPath: "/rootfs"

Add the following lines under volumes:

 - name: kernel-modules
   hostPath:
   path: "/usr/lib/modules"
 - name: host-rootfs
   hostPath:
   path: "/"

Replace the value under IMAGE_VERSION with v3.11.5 or v3.11.8 depending on the version you want to upgrade to.

- displayName: heketi container image name
  name: IMAGE_NAME
  required: true
  value: registry.redhat.io/rhgs3/rhgs-volmanager-rhel7
- displayName: heketi container image version
  name: IMAGE_VERSION
  required: true
  value: v3.11.8

Execute the following commands to create the gluster DaemonSet:
```
# oc process glusterfs | oc create -f -
```
For example,
```
# oc process glusterfs | oc create -f -
Deamonset “glusterfs” created
```
Note
If a cluster has more than 1000 volumes refer to How to change the default PVS limit in Openshift Container Storage and add the required parameters before proceeding with the upgrade.

Execute the following command to identify the old gluster pods that needs to be deleted:

# oc get pods

For example,

# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d

Execute the following command and ensure that the bricks are not more than 90% full:
```
# df -kh | grep -v ^Filesystem | awk '{if(int($5)>90) print $0}'
```
Note
If the bricks are close to 100% utilization, then the Logical Volume Manager(LVM) activation for these bricks may take a long time or can get stuck once the pod or node is rebooted. It is advised to bring down the utilization of that brick or expand the physical volume(PV) that is using the logical volume(LV).
Note
The df command is not applicable to bricks that belong to Block Hosting Volume(BHV). On a BHV , the used size of the bricks generated by the df command is the added size of the blockvolumes of that Gluster volume, it is not the size of the data that resides in the blockvolumes. For more information refer to How To Identify Block Volumes and Block Hosting Volumes in Openshift Container Storage.
Execute the following command to delete the old gluster pods. Gluster pods should follow rolling upgrade. Hence, you must ensure that the new pod is running before deleting the next old gluster pod. We support OnDelete Strategy DaemonSet update strategy. With OnDelete Strategy update strategy, after you update a DaemonSet template, new DaemonSet pods will only be created when you manually delete old DaemonSet pods.
1. To delete the old gluster pods, execute the following command:
```
# oc delete pod <gluster_pod>
```
  For example,
```
# oc delete pod glusterfs-0vcf3
pod  “glusterfs-0vcf3” deleted
```
  Note
  Before deleting the next pod, self heal check has to be made:
  Run the following command to access shell on gluster pod:
  # oc rsh <gluster_pod_name>
  Run the following command to check the self-heal status of all the volumes:
  # for eachVolume in $(gluster volume list); do gluster volume heal $eachVolume info ; done | grep "Number of entries: [^0]$"
2. The delete pod command will terminate the old pod and create a new pod. Run # oc get pods -w and check the Age of the pod and READY status should be 1/1. The following is the example output showing the status progression from termination to creation of the pod.
```
# oc get pods -w
  NAME                             READY     STATUS        RESTARTS   AGE
  glusterfs-0vcf3                  1/1       Terminating   0          3d
…
```

Execute the following command to verify that the pods are running:

# oc get pods

For example,

# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d

Execute the following command to verify if you have upgraded the pod to the latest version:

# oc rsh <gluster_pod_name> glusterd --version

For example:

 # oc rsh glusterfs-4cpcc glusterd --version
glusterfs 6.0

Check the Red Hat Gluster Storage op-version by executing the following command on one of the gluster pods.
```
# gluster vol get all cluster.op-version
```
After you upgrade the Gluster pods, ensure that you set Heketi back to operational mode:
- Scale up the DC (Deployment Configuration).
```
# oc scale dc <heketi_dc> --replicas=1
```

Set the cluster.op-version to 70200 on any one of the pods:

Important

Ensure all the gluster pods are updated before changing the cluster.op-version.

# gluster --timeout=3600 volume set all cluster.op-version 70200

Execute the following steps to enable server.tcp-user-timeout on all volumes.

Note

The "server.tcp-user-timeout" option specifies the maximum amount of the time (in seconds) the transmitted data from the application can remain unacknowledged from the brick.

It is used to detect force disconnections and dead connections (if a node dies unexpectedly, a firewall is activated, etc.,) early and make it possible for applications to reduce the overall failover time.

List the glusterfs pod using the following command:

# oc get pods

For example:

# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d

Remote shell into one of the glusterfs pods. For example:
```
# oc rsh glusterfs-0vcf3
```

Execute the following command:

# for eachVolume in `gluster volume list`; do echo $eachVolume; gluster volume set $eachVolume server.tcp-user-timeout 42 ; done

For example:

# for eachVolume in `gluster volume list`; do echo $eachVolume; gluster volume set $eachVolume server.tcp-user-timeout 42 ; done
  volume1
  volume set: success
  volume2
volume set: success

If a gluster-block-provisoner-pod already exists then delete it by executing the following commands:
```
# oc delete dc glusterblock-provisioner-dc
```
For example:
```
# oc delete dc glusterblock-storage-provisioner-dc
```

Delete the following resources from the old pod:

# oc delete clusterroles.authorization.openshift.io glusterblock-provisioner-runner
# oc delete serviceaccounts glusterblock-provisioner
serviceaccount "glusterblock-provisioner" deleted
# oc delete clusterrolebindings.authorization.openshift.io glusterblock-provisioner

Execute the following commands to deploy the gluster-block provisioner:

`sed -e 's/${NAMESPACE}/<NAMESPACE>/' /usr/share/heketi/templates/glusterblock-provisioner.yaml | sed -e 's/<VERSION>/<NEW-VERSION>/' | oc create -f -

<VERSION>

Existing version of OpenShift Container Storage.

<NEW-VERSION>

Either 3.11.5 or 3.11.8 depending on the version you are upgrading to.

# oc adm policy add-cluster-role-to-user glusterblock-provisioner-runner system:serviceaccount:<NAMESPACE>:glusterblock-provisioner

For example:

`sed -e 's/${NAMESPACE}/storage-project/' /usr/share/heketi/templates/glusterblock-provisioner.yaml | sed -e 's/3.11.4/3.11.8/' | oc create -f -

# oc adm policy add-cluster-role-to-user glusterblock-provisioner-runner system:serviceaccount:storage-project:glusterblock-provisioner

Brick multiplexing is a feature that allows adding multiple bricks into one process. This reduces resource consumption and allows us to run more bricks than before with the same memory consumption. It is enabled by default from Container-Native Storage 3.6 onward. During an upgrade from Container-Native Storage 3.10 to Red Hat Openshift Container Storage 3.11, to turn brick multiplexing on, execute the following commands:
1. To exec into the Gluster pod, execute the following command and rsh into any of the gluster pods:
```
# oc rsh <gluster_pod_name>
```
2. Verify the brick multiplex status:
```
# gluster v get all all
```
3. If it is disabled, then execute the following command to enable brick multiplexing:
  Note
  Ensure that all volumes are in a stop state or no bricks are running while brick multiplexing is enabled.
  # gluster volume set all cluster.brick-multiplex on
  For example:
```
# oc rsh glusterfs-770ql

  sh-4.2# gluster volume set all cluster.brick-multiplex on
  Brick-multiplexing is supported only for container workloads (Independent or Converged mode). Also it is advised to make sure that either all volumes are in stopped state or no bricks are running before this option is modified.Do you still want to continue? (y/n) y
volume set: success
```
4. List all the volumes in the trusted storage pool. This step is only required if the volume set operation is performed:
  For example:
```
# gluster volume list

  heketidbstorage
  vol_194049d2565d2a4ad78ef0483e04711e
  ...
  ...
```
  Restart all the volumes. This step is only required if the volume set operation is performed along with the previous step:
```
# gluster vol stop <VOLNAME>
# gluster vol start <VOLNAME>
```
Support for S3 compatible Object Store in Red Hat Openshift Container Storage is under technology preview. To enable S3 compatible object store, see https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html/operations_guide/s3_object_store.

Note

If you have glusterfs registry pods, then proceed with the steps listed in Section 6.2, “Upgrading the pods in the glusterfs registry group” to upgrade heketi and glusterfs registry pods.
If you do not have glusterfs registry pods, then proceed with the steps listed in ] to bring back your heketi pod and then proceed with the steps listed in xref:chap-upgrade_client_common[ to upgrade the client on Red Hat Openshift Container Platform Nodes.

6.1.4. Upgrading if existing version deployed by using Ansible

6.1.4.1. Upgrading Heketi Server

The following commands must be executed on the client machine.

Execute the following steps to check for any pending Heketi operatons:
1. Execute the following command to access your project:
```
# oc project <project_name>
```
  For example:
```
# oc project storage-project
```
2. Wait for the ongoing operations to complete and execute the following command to monitor if there are any ongoing operations:
```
# heketi-cli server operations info
```
Backup the Heketi database file.
```
# heketi-cli db dump > heketi-db-dump-$(date -I).json
```
Note
The json file created can be used to restore and therefore should be stored in persistent storage of your choice.
Execute the following command to update the heketi client packages. Update the heketi-client package on all the OCP nodes where it is installed. Newer installations may not have the heketi-client rpm installed on any OCP nodes:
```
# yum update heketi-client -y
```
Execute the following command to get the current HEKETI_ADMIN_KEY.
The OCS admin can choose to set any phrase for user key as long as it is not used by their infrastructure. It is not used by any of the OCS default installed resources.
```
# oc get secret heketi-storage-admin-secret -o jsonpath='{.data.key}'|base64 -d;echo
```
If the HEKETI_USER_KEY was set previously, you can obtain it by using the following command:
```
# oc describe pod <heketi-pod>
```
Execute the following command to delete the heketi template.
```
# oc delete templates heketi
```

Execute the following command to install the heketi template.

# oc create -f /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/heketi-template.yml
template "heketi" created

Execute the following step to edit the template:

# oc get templates
NAME			  DESCRIPTION		     PARAMETERS		OBJECTS
glusterblock-provisioner  glusterblock provisioner   3 (2 blank)	4
			  template
glusterfs		  GlusterFS DaemonSet 	     5 (1 blank)	1
			  template
heketi			  Heketi service deployment  7 (3 blank)	3
template

If the existing template has IMAGE_NAME and IMAGE_VERSION as two parameters, then edit the template to change the HEKETI_USER_KEY, HEKETI_ADMIN_KEY, HEKETI_ROUTE, IMAGE_NAME, IMAGE_VERSION, CLUSTER_NAME and HEKETI_LVM_WRAPPER as shown in the example below.

# oc edit template heketi
parameters:
  - description: Set secret for those creating volumes as type user
  displayName: Heketi User Secret
  name: HEKETI_USER_KEY
  value: <heketiuserkey>
  - description: Set secret for administration of the Heketi service as user admin
  displayName: Heketi Administrator Secret
  name: HEKETI_ADMIN_KEY
  value: <adminkey>
  - description: Set the executor type, kubernetes or ssh
  displayName: heketi executor type
  name: HEKETI_EXECUTOR
  value: kubernetes
  - description: Set the hostname for the route URL
  displayName: heketi route name
  name: HEKETI_ROUTE
  value: heketi-storage
  - displayName: heketi container image name
  name: IMAGE_NAME
  required: true
  value: registry.redhat.io/rhgs3/rhgs-volmanager-rhel7
  - displayName: heketi container image version
  name: IMAGE_VERSION
  required: true
  value: v3.11.8
  - description: A unique name to identify this heketi service, useful for running
  multiple heketi instances
  displayName: GlusterFS cluster name
  name: CLUSTER_NAME
  value: storage
  - description: Heketi can use a wrapper to execute LVM commands, i.e. run commands in the host namespace instead of in the Gluster container
  name: HEKETI_LVM_WRAPPER
  displayName: Wrapper for executing LVM commands
  value: /usr/sbin/exec-on-host

If the template has only IMAGE_NAME, then edit the template to change the HEKETI_USER_KEY, HEKETI_ADMIN_KEY, HEKETI_ROUTE, IMAGE_NAME, CLUSTER_NAME and HEKETI_LVM_WRAPPER as shown in the example below.

# oc edit template heketi
parameters:
- description: Set secret for those creating volumes as type user
  displayName: Heketi User Secret
  name: HEKETI_USER_KEY
  value: <heketiuserkey>
- description: Set secret for administration of the Heketi service as user admin
  displayName: Heketi Administrator Secret
  name: HEKETI_ADMIN_KEY
  value: <adminkey>
- description: Set the executor type, kubernetes or ssh
  displayName: heketi executor type
  name: HEKETI_EXECUTOR
  value: kubernetes
- description: Set the hostname for the route URL
  displayName: heketi route name
  name: HEKETI_ROUTE
  value: heketi-storage
- displayName: heketi container image name
  name: IMAGE_NAME
  required: true
  value: registry.redhat.io/rhgs3/rhgs-volmanager-rhel7:v3.11.8
- description: A unique name to identify this heketi service, useful for running
   multiple heketi instances
  displayName: GlusterFS cluster name
  name: CLUSTER_NAME
  value: storage
- description: Heketi can use a wrapper to execute LVM commands, i.e. run commands in the host namespace instead of in the Gluster container
  name: HEKETI_LVM_WRAPPER
  displayName: Wrapper for executing LVM commands
  value: /usr/sbin/exec-on-host

Note

If a cluster has more than 1000 volumes refer to How to change the default PVS limit in Openshift Container Storage and add the required parameters before proceeding with the upgrade.

Execute the following command to delete the deployment configuration, service, and route for heketi:
Note
The names of these parameters can be referenced from output of the following command:
```
# oc get all | grep heketi
```
```
# oc delete deploymentconfig,service,route heketi-storage
```
Execute the following command to deploy the Heketi service, route, and deployment configuration which will be used to create persistent volumes for OpenShift:
```
# oc process heketi | oc create -f -

service "heketi" created
route "heketi" created
deploymentconfig "heketi" created
```
Note
It is recommended that the heketidbstorage volume be tuned for db workloads. Newly installed Openshift Container Storage deployments tune the heketidbstorage volume automatically. For older deployments, follow the KCS article Planning to run containerized DB or nosql workloads on Openshift Container Storage? and perform the volume set operation for the volume heketidbstorage.

Execute the following command to verify that the containers are running:

# oc get pods

For example:

# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d

6.1.4.2. Upgrading the Red Hat Gluster Storage Pods

The following commands must be executed on the client machine.

Following are the steps for updating a DaemonSet for glusterfs:

Execute the following steps to stop the Heketi pod to prevent it from accepting any new request for volume creation or volume deletion:
1. Execute the following command to access your project:
```
# oc project <project_name>
```
  For example:
```
# oc project storage-project
```
2. Execute the following command to get the DeploymentConfig:
```
# oc get dc
```
3. Execute the following command to set heketi server to accept requests only from the local-client:
```
# heketi-cli server mode set local-client
```
4. Wait for the ongoing operations to complete and execute the following command to monitor if there are any ongoing operations:
```
# heketi-cli server operations info
```
5. Execute the following command to reduce the replica count from 1 to 0. This brings down the Heketi pod:
```
# oc scale dc <heketi_dc> --replicas=0
```
6. Execute the following command to verify that the heketi pod is no longer present:
```
# oc get pods
```
Execute the following command to find the DaemonSet name for gluster
```
# oc get ds
```
Execute the following command to delete the DaemonSet:
```
# oc delete ds <ds-name> --cascade=false
```
Using --cascade=false option while deleting the old DaemonSet does not delete the gluster pods but deletes only the DaemonSet. After deleting the old DaemonSet, you must load the new one. When you manually delete the old pods, the new pods which are created will have the configurations of the new DaemonSet.
For example,
```
# oc delete ds glusterfs-storage --cascade=false
daemonset "glusterfs-storage" deleted
```

Execute the following commands to verify all the old pods are up:

# oc get pods

For example,

# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d

Execute the following command to delete the old glusterfs template.
```
# oc delete templates glusterfs
```

Execute the following command to register new glusterfs template.

# oc create -f /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/glusterfs-template.yml
template "glusterfs" created

Execute the following command to edit the old glusterfs template.

# oc get templates
NAME			  DESCRIPTION		     PARAMETERS		OBJECTS
glusterblock-provisioner  glusterblock provisioner   3 (2 blank)	4
        template
glusterfs		  GlusterFS DaemonSet 	     5 (1 blank)	1
        template
heketi			  Heketi service deployment  7 (3 blank)	3
template

If the template has IMAGE_NAME and IMAGE_VERSION as two separate parameters, then update the glusterfs template as following. For example:

# oc edit template glusterfs
- displayName: GlusterFS container image name
	  name: IMAGE_NAME
	  required: true
	  value: registry.redhat.io/rhgs3/rhgs-server-rhel7
- displayName: GlusterFS container image version
	  name: IMAGE_VERSION
	  required: true
	  value: v3.11.8
- description: A unique name to identify which heketi service manages this cluster, useful for running
     multiple heketi instances
  displayName: GlusterFS cluster name
  name: CLUSTER_NAME
  value: storage

Note

If a cluster has more than 1000 volumes refer to How to change the default PVS limit in Openshift Container Storage and add the required parameters before proceeding with the upgrade.

If the template has only IMAGE_NAME as a parameter, then update the glusterfs template as following. For example:

# oc edit template glusterfs
- displayName: GlusterFS container image name
  name: IMAGE_NAME
  required: true
  value: registry.redhat.io/rhgs3/rhgs-server-rhel7:v3.11.8
- description: A unique name to identify which heketi service manages this cluster, useful for running
     multiple heketi instances
  displayName: GlusterFS cluster name
  name: CLUSTER_NAME
value: storage

Note

Ensure that the CLUSTER_NAME variable is set to the correct value

Label all the OpenShift Container Platform nodes that has the Red Hat Gluster Storage pods:
1. Check if the nodes are labelled with the appropriate label by using the following command:
```
# oc get nodes -l glusterfs=storage-host
```

Execute the following commands to create the gluster DaemonSet:

# oc process glusterfs | oc create -f -

For example,

# oc process glusterfs | oc create -f -
Deamonset “glusterfs” created

Execute the following command to identify the old gluster pods that needs to be deleted:

# oc get pods

For example,

# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d

Execute the following command and ensure that the bricks are not more than 90% full:
```
# df -kh | grep -v ^Filesystem | awk '{if(int($5)>90) print $0}'
```
Note
If the bricks are close to 100% utilization, then the Logical Volume Manager(LVM) activation for these bricks may take a long time or can get stuck once the pod or node is rebooted. It is advised to bring down the utilization of that brick or expand the physical volume(PV) that is using the logical volume(LV).
Note
The df command is not applicable to bricks that belong to Block Hosting Volume(BHV). On a BHV , the used size of the bricks generated by the df command is the added size of the blockvolumes of that Gluster volume, it is not the size of the data that resides in the blockvolumes. For more information refer to How To Identify Block Volumes and Block Hosting Volumes in Openshift Container Storage.
Execute the following command to delete the old gluster pods. Gluster pods should follow rolling upgrade. Hence, you must ensure that the new pod is running before deleting the next old gluster pod. We support OnDelete Strategy DaemonSet update strategy. With OnDelete Strategy update strategy, after you update a DaemonSet template, new DaemonSet pods will only be created when you manually delete old DaemonSet pods.
1. To delete the old gluster pods, execute the following command:
```
# oc delete pod <gluster_pod>
```
  For example,
```
# oc delete pod glusterfs-0vcf3
pod  “glusterfs-0vcf3” deleted
```
  Note
  Before deleting the next pod, self heal check has to be made:
  Run the following command to access shell on gluster pod:
  # oc rsh <gluster_pod_name>
  Run the following command to check the self-heal status of all the volumes:
  # for eachVolume in $(gluster volume list); do gluster volume heal $eachVolume info ; done | grep "Number of entries: [^0]$"
2. The delete pod command will terminate the old pod and create a new pod. Run # oc get pods -w and check the Age of the pod and READY status should be 1/1. The following is the example output showing the status progression from termination to creation of the pod.
```
# oc get pods -w
NAME                             READY     STATUS        RESTARTS   AGE
glusterfs-0vcf3                  1/1       Terminating   0          3d
…
```

Execute the following command to verify that the pods are running:

# oc get pods

For example,

# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d

Execute the following command to verify if you have upgraded the pod to the latest version:

# oc rsh <gluster_pod_name> glusterd --version

For example:

# oc rsh glusterfs-4cpcc glusterd --version
glusterfs 6.0

Check the Red Hat Gluster Storage op-version by executing the following command on one of the gluster pods.
```
# gluster vol get all cluster.op-version
```
After you upgrade the Gluster pods, ensure that you set Heketi back to operational mode:
- Scale up the DC (Deployment Configuration).
```
# oc scale dc <heketi_dc> --replicas=1
```
Set the cluster.op-version to 70200 on any one of the pods:
Note
Ensure all the gluster pods are updated before changing the cluster.op-version.
```
# gluster --timeout=3600 volume set all cluster.op-version 70200
```

Execute the following steps to enable server.tcp-user-timeout on all volumes.

Note

The "server.tcp-user-timeout" option specifies the maximum amount of the time (in seconds) the transmitted data from the application can remain unacknowledged from the brick.

List the glusterfs pod using the following command:

# oc get pods

For example:

# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d

Remote shell into one of the glusterfs pods. For example:
```
# oc rsh glusterfs-0vcf3
```

Execute the following command:

# for eachVolume in `gluster volume list`; do echo $eachVolume; gluster volume set $eachVolume server.tcp-user-timeout 42 ; done

For example:

# for eachVolume in `gluster volume list`; do echo $eachVolume; gluster volume set $eachVolume server.tcp-user-timeout 42 ; done
    volume1
    volume set: success
    volume2
volume set: success

If a gluster-block-provisoner-pod already exists then delete it by executing the following commands:
```
# oc delete dc glusterblock-provisioner-dc
```
For example:
```
# oc delete dc glusterblock-storage-provisioner-dc
```
Execute the following command to delete the old glusterblock provisioner template.
```
 # oc delete templates glusterblock-provisioner
```

Create a glusterblock provisioner template. For example:

# oc create -f /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/glusterblock-provisioner.yml
template.template.openshift.io/glusterblock-provisioner created

Depending on the OCP version, edit the glusterblock-provisioner template to change the IMAGE_NAME, IMAGE_VERSION and NAMESPACE.

# oc get templates
NAME			  DESCRIPTION		     PARAMETERS		OBJECTS
glusterblock-provisioner  glusterblock provisioner   3 (2 blank)	4
        template
glusterfs		  GlusterFS DaemonSet 	     5 (1 blank)	1
        template
heketi			  Heketi service deployment  7 (3 blank)	3
template

If the template has IMAGE_NAME and IMAGE_VERSION as two separate parameters, then update the glusterblock-provisioner template as following. For example:

# oc edit template glusterblock-provisioner
- displayName: glusterblock provisioner container image name
name: IMAGE_NAME
required: true
value: registry.redhat.io/rhgs3/rhgs-gluster-block-prov-rhel7
- displayName: glusterblock provisioner container image version
name: IMAGE_VERSION
required: true
value: v3.11.8
- description: The namespace in which these resources are being created
displayName: glusterblock provisioner namespace
name: NAMESPACE
required: true
value: glusterfs
- description: A unique name to identify which heketi service manages this cluster,
  useful for running multiple heketi instances
displayName: GlusterFS cluster name
name: CLUSTER_NAME
value: storage

If the template has only IMAGE_NAME as a parameter, then update the glusterblock-provisioner template as following. For example:

# oc edit template glusterblock-provisioner
- displayName: glusterblock provisioner container image name
name: IMAGE_NAME
required: true
value: registry.redhat.io/rhgs3/rhgs-gluster-block-prov-rhel7:v3.11.8
- description: The namespace in which these resources are being created
displayName: glusterblock provisioner namespace
name: NAMESPACE
required: true
value: glusterfs
- description: A unique name to identify which heketi service manages this cluster,
            useful for running multiple heketi instances
displayName: GlusterFS cluster name
name: CLUSTER_NAME
value: storage

Delete the following resources from the old pod

# oc delete clusterroles.authorization.openshift.io glusterblock-provisioner-runner
# oc delete serviceaccounts glusterblock-storage-provisioner
# oc delete clusterrolebindings.authorization.openshift.io glusterblock-storage-provisioner

Before running oc process determine the correct provisioner name. If there are more than one gluster block provisioner running in your cluster the names must differ from all other provisioners.
For example,
- If there are 2 or more provisioner the name should be gluster.org/glusterblock-<namespace> where, namespace is replaced by the namespace that the provisioner is deployed in.
- If there is only one provisioner, installed prior to 3.11.8, gluster.org/glusterblock is sufficent. If the name currently in use already has a unique namespace suffix, reuse the existing name.

After editing the template, execute the following command to create the deployment configuration:

# oc process glusterblock-provisioner -o yaml | oc create -f -

For example:

# oc process glusterblock-provisioner -o yaml | oc create -f -
clusterrole.authorization.openshift.io/glusterblock-provisioner-runner created
serviceaccount/glusterblock-storage-provisioner created
clusterrolebinding.authorization.openshift.io/glusterblock-storage-provisioner created
deploymentconfig.apps.openshift.io/glusterblock-storage-provisioner-dc created

Brick multiplexing is a feature that allows adding multiple bricks into one process. This reduces resource consumption and allows us to run more bricks than before with the same memory consumption. It is enabled by default from Container-Native Storage 3.6 onward. During an upgrade from Container-Native Storage 3.10 to Red Hat Openshift Container Storage 3.11, to turn brick multiplexing on, execute the following commands:
1. To exec into the Gluster pod, execute the following command and rsh into any of the gluster pods:
```
# oc rsh <gluster_pod_name>
```
2. Verify the brick multiplex status:
```
# gluster v get all all
```
3. If it is disabled, then execute the following command to enable brick multiplexing:
  Note
  Ensure that all volumes are in a stop state or no bricks are running while brick multiplexing is enabled.
  # gluster volume set all cluster.brick-multiplex on
  For example:
```
# oc rsh glusterfs-770ql
sh-4.2# gluster volume set all cluster.brick-multiplex on
Brick-multiplexing is supported only for container workloads (Independent or Converged mode). Also it is advised to make sure that either all volumes are in stopped state or no bricks are running before this option is modified.Do you still want to continue? (y/n) y
volume set: success
```
4. List all the volumes in the trusted storage pool. This step is only required if the volume set operation is performed:
  For example:
```
# gluster volume list

heketidbstorage
vol_194049d2565d2a4ad78ef0483e04711e
...
...
```
  Restart all the volumes. This step is only required if the volume set operation is performed along with the previous step:
```
# gluster vol stop <VOLNAME>
# gluster vol start <VOLNAME>
```
Support for S3 compatible Object Store in Red Hat Openshift Container Storage is under technology preview. To enable S3 compatible object store, see https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html/operations_guide/s3_object_store.
Note
- If you have glusterfs registry pods, then proceed with the steps listed in Section 6.2, “Upgrading the pods in the glusterfs registry group” to upgrade heketi and glusterfs registry pods.
- If you do not have glusterfs registry pods, then proceed with the steps listed in ] to bring back your heketi pod and then proceed with the steps listed in xref:chap-upgrade_client_common[ to upgrade the client on Red Hat Openshift Container Platform Nodes.

All storage classes that use gluster block volume provisioning must match exactly to one of the provisioner names in the cluster. To check the list of storage classes that refer to a block provisioner, in a given namespace, run the following command:

# oc get sc -o custom-columns=NAME:.metadata.name,PROV:.provisioner,RSNS:.parameters.restsecretnamespace | grep 'gluster.org/glusterblock' | grep <namespace>

Example:

# oc get sc -o custom-columns=NAME:.metadata.name,PROV:.provisioner,RSNS:.parameters.restsecretnamespace | grep 'gluster.org/glusterblock' | grep app-storage
glusterfs-storage-block   gluster.org/glusterblock-app-storage   app-storage

Check each storage class provisioner name, if it does not match the block provisioner name configured for that namespace it must be updated. If the block provisioner name already matches the configured provisioner name, nothing else needs to be done. Use the list generated above and include all storage class names where the provionser name must be updated.
For every storage class in this list do the following:

# oc get sc  -o yaml <storageclass>  > storageclass-to-edit.yaml
# oc delete sc  <storageclass>
# sed 's,gluster.org/glusterblock$,gluster.org/glusterblock-<namespace>,' storageclass-to-edit.yaml | oc create -f -

Example:

# oc get sc  -o yaml gluster-storage-block  > storageclass-to-edit.yaml
# oc delete sc  gluster-storage-block
# sed 's,gluster.org/glusterblock$,gluster.org/glusterblock-app-storage,' storageclass-to-edit.yaml | oc create -f -

6.2. Upgrading the pods in the glusterfs registry group

The following sections provide steps to upgrade your glusterfs registry pods.

6.2.1. Prerequisites

Ensure the following prerequisites are met:

Section 3.1.3, “Red Hat OpenShift Container Platform and Red Hat Openshift Container Storage Requirements”
Ensure to have the supported versions of OpenShift Container Platform with Red Hat Gluster Storage Server and Red Hat Openshift Container Storage. For more information on supported versions, see Section 3.1.1, “Supported Versions”
Ensure to run the following command to get the latest versions of Ansible templates.
```
# yum update openshift-ansible
```

Note

For deployments using cns-deploy tool, the templates are available in the following location:

gluster template - /usr/share/heketi/templates/glusterfs-template.yaml
heketi template - /usr/share/heketi/templates/heketi-template.yaml
glusterblock-provisioner template - /usr/share/heketi/templates/glusterblock-provisioner.yaml

For deployments using ansible playbook the templates are available in the following location:

gluster template - /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/glusterfs-template.yml
heketi template - /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/heketi-template.yml
glusterblock-provisioner template - /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/glusterblock-provisioner.yml

6.2.2. Restoring original label values for /dev/log

Note

Follow this procedure only if you are upgrading your environment from Red Hat Container Native Storage 3.9 to Red Hat Openshift Container Storage 3.11.8.

Skip this procedure if you are upgrading your environment from Red Hat Openshift Container Storage 3.10 and above to Red Hat Openshift Container Storage 3.11.8.

To restore the original selinux label, execute the following commands:

Create a directory and soft links on all nodes that run gluster pods:

# mkdir /srv/<directory_name>
# cd /srv/<directory_name>/   # same dir as above
# ln -sf /dev/null systemd-tmpfiles-setup-dev.service
# ln -sf /dev/null systemd-journald.service
# ln -sf /dev/null systemd-journald.socket

Edit the daemonset that creates the glusterfs pods on the node which has oc client:

# oc edit daemonset <daemonset_name>

Under volumeMounts section add a mapping for the volume:

- mountPath: /usr/lib/systemd/system/systemd-journald.service
  name: systemd-journald-service
- mountPath: /usr/lib/systemd/system/systemd-journald.socket
  name: systemd-journald-socket
- mountPath: /usr/lib/systemd/system/systemd-tmpfiles-setup-dev.service
name: systemd-tmpfiles-setup-dev-service

Under volumes section add a new host path for each service listed:

Note

The path mentioned in here should be the same as mentioned in Step 1.

- hostPath:
   path: /srv/<directory_name>/systemd-journald.socket
   type: ""
  name: systemd-journald-socket
- hostPath:
   path: /srv/<directory_name>/systemd-journald.service
   type: ""
  name: systemd-journald-service
- hostPath:
   path: /srv/<directory_name>/systemd-tmpfiles-setup-dev.service
   type: ""
name: systemd-tmpfiles-setup-dev-service

Run the following command on all nodes that run gluster pods. This will reset the label:
```
# restorecon /dev/log
```

Execute the following command to check the status of self heal for all volumes:

# oc rsh <gluster_pod_name>
# for each_volume in `gluster volume list`; do gluster volume heal $each_volume info ; done  | grep  "Number of entries: [^0]$"

Wait for self-heal to complete.

Execute the following command and ensure that the bricks are not more than 90% full:
```
# df -kh | grep -v ^Filesystem | awk '{if(int($5)>90) print $0}'
```
Note
If the bricks are close to 100% utilization, then the Logical Volume Manager(LVM) activation for these bricks may take a long time or can get stuck once the pod or node is rebooted. It is advised to bring down the utilization of that brick or expand the physical volume(PV) that is using the logical volume(LV).
Note
The df command is not applicable to bricks that belong to Block Hosting Volume(BHV). On a BHV , the used size of the bricks generated by the df command is the added size of the blockvolumes of that Gluster volume, it is not the size of the data that resides in the blockvolumes. For more information refer to How To Identify Block Volumes and Block Hosting Volumes in Openshift Container Storage.
Execute the following command on any one of the gluster pods to set the maximum number of bricks (250) that can run on a single instance of glusterfsd process:
```
# gluster volume set all cluster.max-bricks-per-process 250
```
1. Execute the following command on any one of the gluster pods to ensure that the option is set correctly:
```
# gluster volume get all cluster.max-bricks-per-process
```
  For example:
```
# gluster volume get all cluster.max-bricks-per-process
cluster.max-bricks-per-process 250
```
Execute the following command on the node which has oc client to delete the gluster pod:
```
# oc delete pod <gluster_pod_name>
```
To verify if the pod is ready, execute the following command:
```
# oc get pods -l glusterfs=registry-pod
```
Login to the node hosting the pod and check the selinux label of /dev/log
```
# ls -lZ /dev/log
```
The output should show devlog_t label
For example:
```
#  ls -lZ /dev/log
srw-rw-rw-. root root system_u:object_r:devlog_t:s0    /dev/log
```
Exit the node.

In the gluster pod, check if the label value is devlog_t:

# oc rsh <gluster_pod_name>
# ls -lZ /dev/log

For example:

#  ls -lZ /dev/log
srw-rw-rw-. root root system_u:object_r:devlog_t:s0    /dev/log

Perform steps 4 to 9 for other pods.

6.2.3. Upgrading if existing version deployed by using cns-deploy

6.2.3.1. Upgrading cns-deploy and Heketi Server

The following commands must be executed on the client machine.

Execute the following command to update the heketi client and cns-deploy packages:
```
# yum update cns-deploy -y
# yum update heketi-client -y
```

Backup the Heketi registry database file

# heketi-cli db dump > heketi-db-dump-$(date -I).json

Execute the following command to delete the heketi template.
```
# oc delete templates heketi
```
Execute the following command to get the current HEKETI_ADMIN_KEY.
The OCS admin can choose to set any phrase for user key as long as it is not used by their infrastructure. It is not used by any of the OCS default installed resources.
```
# oc get secret <heketi-admin-secret-name> -o jsonpath='{.data.key}'|base64 -d;echo
```

Execute the following command to install the heketi template.

# oc create -f /usr/share/heketi/templates/heketi-template.yaml
template "heketi" created

Execute the following command to grant the heketi Service Account the necessary privileges.
```
# oc policy add-role-to-user edit system:serviceaccount:<project_name>:heketi-service-account
# oc adm policy add-scc-to-user privileged -z heketi-service-account
```
For example,
```
# oc policy add-role-to-user edit system:serviceaccount:storage-project:heketi-service-account
# oc adm policy add-scc-to-user privileged -z heketi-service-account
```
Note
The service account used in heketi pod needs to be privileged because Heketi/rhgs-volmanager pod mounts the heketidb storage Gluster volume as a "glusterfs" volume type and not as a PersistentVolume (PV).
As per the security-context-constraints regulations in OpenShift, ability to mount volumes which are not of the type configMap, downwardAPI, emptyDir, hostPath, nfs, persistentVolumeClaim, secret is granted only to accounts with privileged Security Context Constraint (SCC).
Execute the following command to generate a new heketi configuration file.
```
# sed -e "s/\${HEKETI_EXECUTOR}/kubernetes/" -e "s#\${HEKETI_FSTAB}#/var/lib/heketi/fstab#" -e "s/\${SSH_PORT}/22/" -e "s/\${SSH_USER}/root/" -e "s/\${SSH_SUDO}/false/" -e "s/\${BLOCK_HOST_CREATE}/true/" -e "s/\${BLOCK_HOST_SIZE}/500/" "/usr/share/heketi/templates/heketi.json.template" > heketi.json
```
- The BLOCK_HOST_SIZE parameter controls the size (in GB) of the automatically created Red Hat Gluster Storage volumes hosting the gluster-block volumes (For more information, see https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html-single/operations_guide/index#Block_Storage). This default configuration will dynamically create block-hosting volumes of 500GB in size as more space is required.
- Alternatively, copy the file /usr/share/heketi/templates/heketi.json.template to heketi.json in the current directory and edit the new file directly, replacing each "${VARIABLE}" string with the required parameter.
  Note
  JSON formatting is strictly required (e.g. no trailing spaces, booleans in all lowercase).
Execute the following command to create a secret to hold the configuration file.
```
# oc create secret generic <heketi-registry-config-secret> --from-file=heketi.json
```
Note
If the heketi-registry-config-secret file already exists, then delete the file and run the following command.
Execute the following command to delete the deployment configuration, service, and route for heketi:
```
# oc delete deploymentconfig,service,route heketi-registry
```

Edit the heketi template.

Edit the HEKETI_USER_KEY and HEKETI_ADMIN_KEY parameters.

# oc edit template heketi
parameters:
- description: Set secret for those creating volumes as type user
  displayName: Heketi User Secret
  name: HEKETI_USER_KEY
  value: <heketiuserkey>
- description: Set secret for administration of the Heketi service as user admin
  displayName: Heketi Administrator Secret
  name: HEKETI_ADMIN_KEY
  value: <adminkey>
- description: Set the executor type, kubernetes or ssh
  displayName: heketi executor type
  name: HEKETI_EXECUTOR
  value: kubernetes
- description: Set the hostname for the route URL
  displayName: heketi route name
  name: HEKETI_ROUTE
  value: heketi-storage
- displayName: heketi container image name
  name: IMAGE_NAME
  required: true
  value: registry.redhat.io/rhgs3/rhgs-volmanager-rhel7
- displayName: heketi container image version
  name: IMAGE_VERSION
  required: true
  value: v3.11.8
- description: A unique name to identify this heketi service, useful for running
    multiple heketi instances
  displayName: GlusterFS cluster name
  name: CLUSTER_NAME
  value: storage

Note

If a cluster has more than 1000 volumes refer to How to change the default PVS limit in Openshift Container Storage and add the required parameters before proceeding with the upgrade.

Add an ENV with the name HEKETI_LVM_WRAPPER and value /usr/sbin/exec-on-host.

- description: Heketi can use a wrapper to execute LVM commands, i.e. run commands
in the host namespace instead of in the Gluster container.
displayName: Wrapper for executing LVM commands
name: HEKETI_LVM_WRAPPER
value: /usr/sbin/exec-on-host

Add an ENV with the name HEKETI_DEBUG_UMOUNT_FAILURES and value true.

- description: When unmounting a brick fails, Heketi will not be able to cleanup the
Gluster volume completely. The main causes for preventing to unmount a brick,
seem to originate from Gluster processes. By enabling this option, the heketi.log
will contain the output of 'lsof' to aid with debugging of the Gluster processes
and help with identifying any files that may be left open.
displayName: Capture more details in case brick unmounting fails
name: HEKETI_DEBUG_UMOUNT_FAILURES
required=true

Add an ENV with the name HEKETI_CLI_USER and value admin.
Add an ENV with the name HEKETI_CLI_KEY and the same value provided for the ENV HEKETI_ADMIN_KEY.

Replace the value under IMAGE_VERSION with v3.11.5 or v3.11.8 depending on the version you want to upgrade to.

- displayName: heketi container image name
  name: IMAGE_NAME
  required: true
  value: registry.redhat.io/rhgs3/rhgs-volmanager-rhel7
- displayName: heketi container image version
  name: IMAGE_VERSION
  required: true
  value: v3.11.8

Execute the following command to deploy the Heketi service, route, and deployment configuration which will be used to create persistent volumes for OpenShift:
```
# oc process heketi | oc create -f -

service "heketi-registry" created
route "heketi-registry" created
deploymentconfig-registry "heketi" created
```
Note
It is recommended that the heketidbstorage volume be tuned for db workloads. Newly installed Openshift Container Storage deployments tune the heketidbstorage volume automatically. For older deployments, follow the KCS article Planning to run containerized DB or nosql workloads on Openshift Container Storage? and perform the volume set operation for the volume heketidbstorage.

Execute the following command to verify that the containers are running:

# oc get pods

For example:

# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d

6.2.3.2. Upgrading the Red Hat Gluster Storage Registry Pods

The following commands must be executed on the client machine. .

Following are the steps for updating a DaemonSet for glusterfs:

Execute the following steps to stop the Heketi pod to prevent it from accepting any new request for volume creation or volume deletion:
1. Execute the following command to access your project:
```
# oc project <project_name>
```
  For example:
```
# oc project storage-project
```
2. Execute the following command to get the DeploymentConfig:
```
# oc get ds
```
3. Execute the following command to set heketi server to accept requests only from the local-client:
```
# heketi-cli server mode set local-client
```
4. Wait for the ongoing operations to complete and execute the following command to monitor if there are any ongoing operations:
```
# heketi-cli server operations info
```
5. Execute the following command to reduce the replica count from 1 to 0. This brings down the Heketi pod:
```
# oc scale dc <heketi_dc> --replicas=0
```
6. Execute the following command to verify that the heketi pod is no longer present:
```
# oc get pods
```
Execute the following command to find the DaemonSet name for gluster
```
# oc get ds
```
Execute the following command to delete the DaemonSet:
```
# oc delete ds <ds-name> --cascade=false
```
Using --cascade=false option while deleting the old DaemonSet does not delete the glusterfs_registry pods but deletes only the DaemonSet. After deleting the old DaemonSet, you must load the new one. When you manually delete the old pods, the new pods which are created will have the configurations of the new DaemonSet.
For example,
```
# oc delete ds glusterfs-registry --cascade=false
daemonset "glusterfs-registry" deleted
```

Execute the following commands to verify all the old pods are up:

# oc get pods

For example,

# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d

Execute the following command to delete the old glusterfs template.

# oc delete templates glusterfs

For example,

# oc delete templates glusterfs
template “glusterfs” deleted

Label all the OpenShift Container Platform nodes that has the Red Hat Gluster Storage pods:
1. Check if the nodes are labelled with the appropriate label by using the following command:
```
# oc get nodes -l glusterfs=registry-host
```

Execute the following command to register new glusterfs template.

# oc create -f /usr/share/heketi/templates/glusterfs-template.yaml

For example,

# oc create -f /usr/share/heketi/templates/glusterfs-template.yaml
template “glusterfs” created

Edit the glusterfs template.

Execute the following command:
```
# oc edit template glusterfs
```

Add the following lines under volume mounts:

 - name: kernel-modules
   mountPath: "/usr/lib/modules"
   readOnly: true
 - name: host-rootfs
   mountPath: "/rootfs"

Add the following lines under volumes:

 - name: kernel-modules
   hostPath:
   path: "/usr/lib/modules"
 - name: host-rootfs
   hostPath:
   path: "/"

Replace the value under IMAGE_VERSION with v3.11.5 or v3.11.8 depending on the version you want to upgrade to.

- displayName: heketi container image name
  name: IMAGE_NAME
  required: true
  value: registry.redhat.io/rhgs3/rhgs-volmanager-rhel7
- displayName: heketi container image version
  name: IMAGE_VERSION
  required: true
  value: v3.11.8

Execute the following commands to create the gluster DaemonSet:
```
# oc process glusterfs | oc create -f -
```
For example,
```
# oc process glusterfs | oc create -f -
Deamonset “glusterfs” created
```
Note
If a cluster has more than 1000 volumes refer to How to change the default PVS limit in Openshift Container Storage and add the required parameters before proceeding with the upgrade.

Execute the following command to identify the old glusterfs_registry pods that needs to be deleted:

# oc get pods

For example,

# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d

Execute the following command and ensure that the bricks are not more than 90% full:
```
# df -kh | grep -v ^Filesystem | awk '{if(int($5)>90) print $0}'
```
Note
If the bricks are close to 100% utilization, then the Logical Volume Manager(LVM) activation for these bricks may take a long time or can get stuck once the pod or node is rebooted. It is advised to bring down the utilization of that brick or expand the physical volume(PV) that is using the logical volume(LV).
Note
The df command is not applicable to bricks that belong to Block Hosting Volume(BHV). On a BHV , the used size of the bricks generated by the df command is the added size of the blockvolumes of that Gluster volume, it is not the size of the data that resides in the blockvolumes. For more information refer to How To Identify Block Volumes and Block Hosting Volumes in Openshift Container Storage.
Execute the following command to delete the old glusterfs-registry pods. glusterfs-registry pods should follow rolling upgrade. Hence, you must ensure that the new pod is running before deleting the next old glusterfs-registry pods. We support OnDelete Strategy DaemonSet update strategy. With OnDelete Strategy update strategy, after you update a DaemonSet template, new DaemonSet pods will only be created when you manually delete old DaemonSet pods.
1. To delete the old glusterfs-registry pods, execute the following command:
```
# oc delete pod <gluster_pod>
```
  For example,
```
# oc delete pod glusterfs-0vcf3
pod  “glusterfs-0vcf3” deleted
```
  Note
  Before deleting the next pod, self heal check has to be made:
  Run the following command to access shell on glusterfs-registry pods:
  # oc rsh <gluster_pod_name>
  Run the following command to check the self-heal status of all the volumes: :
  # for eachVolume in $(gluster volume list); do gluster volume heal $eachVolume info ; done | grep "Number of entries: [^0]$"
2. The delete pod command will terminate the old pod and create a new pod. Run # oc get pods -w and check the Age of the pod and READY status should be 1/1. The following is the example output showing the status progression from termination to creation of the pod.
```
# oc get pods -w
NAME                             READY     STATUS        RESTARTS   AGE
glusterfs-0vcf3                  1/1       Terminating   0          3d
…
```

Execute the following command to verify that the pods are running:

# oc get pods

For example,

# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d

Execute the following commands to verify if you have upgraded the pod to the latest version:

# oc rsh <gluster_registry_pod_name> glusterd --version

For example:

 # oc rsh glusterfs-registry-4cpcc glusterd --version
glusterfs 6.0

# rpm -qa|grep gluster

Check the Red Hat Gluster Storage op-version by executing the following command on one of the glusterfs-registry pods.
```
# gluster vol get all cluster.op-version
```
After you upgrade the Gluster pods, ensure that you set Heketi back to operational mode:
- Scale up the DC (Deployment Configuration).
```
# oc scale dc <heketi_dc> --replicas=1
```
Set the cluster.op-version to 70200 on any one of the pods:
Note
Ensure all the glusterfs-registry pods are updated before changing the cluster.op-version.
```
# gluster volume set all cluster.op-version 70200
```

Execute the following steps to enable server.tcp-user-timeout on all volumes.

Note

The "server.tcp-user-timeout" option specifies the maximum amount of the time (in seconds) the transmitted data from the application can remain unacknowledged from the brick.

List the glusterfs pod using the following command:

# oc get pods

For example:

# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d

Remote shell into one of the glusterfs-registry pods. For example:
```
# oc rsh glusterfs-registry-g6vd9
```

Execute the following command:

# for eachVolume in `gluster volume list`; do echo $eachVolume; gluster volume set $eachVolume server.tcp-user-timeout 42 ; done

For example:

# for eachVolume in `gluster volume list`; do echo $eachVolume; gluster volume set $eachVolume server.tcp-user-timeout 42 ; done
volume1
volume set: success
volume2
volume set: success

If a gluster-block-registry-provisoner-pod already exists then delete it by executing the following commands:
```
# oc delete dc <gluster-block-registry-dc>
```
For example:
```
# oc delete dc glusterblock-registry-provisioner-dc
```

Delete the following resources from the old pod

# oc delete clusterroles.authorization.openshift.io glusterblock-provisioner-runner
# oc delete serviceaccounts glusterblock-provisioner
serviceaccount "glusterblock-provisioner" deleted
# oc delete clusterrolebindings.authorization.openshift.io glusterblock-provisioner

Execute the following commands to deploy the gluster-block provisioner:

`sed -e 's/${NAMESPACE}/<NAMESPACE>/' /usr/share/heketi/templates/glusterblock-provisioner.yaml | sed -e 's/<VERSION>/<NEW-VERSION>/' | oc create -f -

<VERSION>

Existing version of OpenShift Container Storage.

<NEW-VERSION>

Either 3.11.5 or 3.11.8 depending on the version you are upgrading to.

# oc adm policy add-cluster-role-to-user glusterblock-provisioner-runner system:serviceaccount:<NAMESPACE>:glusterblock-provisioner

For example:

`sed -e 's/${NAMESPACE}/storage-project/' /usr/share/heketi/templates/glusterblock-provisioner.yaml | sed -e 's/3.11.4/3.11.8/' | oc create -f -

# oc adm policy add-cluster-role-to-user glusterblock-provisioner-runner system:serviceaccount:storage-project:glusterblock-provisioner

Brick multiplexing is a feature that allows adding multiple bricks into one process. This reduces resource consumption and allows us to run more bricks than before with the same memory consumption. It is enabled by default from Container-Native Storage 3.6 onward. During an upgrade from Container-Native Storage 3.10 to Red Hat Openshift Container Storage 3.11, to turn brick multiplexing on, execute the following commands:
1. To exec into the Gluster pod, execute the following command and rsh into any of the gluster pods:
```
# oc rsh <gluster_pod_name>
```
2. Verify the brick multiplex status:
```
# gluster v get all all
```
3. If it is disabled, then execute the following command to enable brick multiplexing:
  Note
  Ensure that all volumes are in a stop state or no bricks are running while brick multiplexing is enabled.
  # gluster volume set all cluster.brick-multiplex on
  For example:
```
# oc rsh glusterfs-registry-g6vd9

sh-4.2# gluster volume set all cluster.brick-multiplex on
Brick-multiplexing is supported only for container workloads (Independent or Converged mode). Also it is advised to make sure that either all volumes are in stopped state or no bricks are running before this option is modified.Do you still want to continue? (y/n) y
volume set: success
```
4. List all the volumes in the trusted storage pool. This step is only required if the volume set operation is performed:
  For example:
```
# gluster volume list

heketidbstorage
vol_194049d2565d2a4ad78ef0483e04711e
...
...
```
  Restart all the volumes. This step is only required if the volume set operation is performed along with the previous step:
```
# gluster vol stop <VOLNAME>
# gluster vol start <VOLNAME>
```
Support for S3 compatible Object Store in Red Hat Openshift Container Storage is under technology preview. To enable S3 compatible object store, see https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html/operations_guide/s3_object_store.
Note
After upgrading the glusterfs registry pods, proceed with the steps listed in ] to bring back your heketi pod and then proceed with the steps listed in xref:chap-upgrade_client_common[ to upgrade the client on Red Hat Openshift Container Platform Nodes.

6.2.4. Upgrading if existing version deployed by using Ansible

6.2.4.1. Upgrading Heketi Server

The following commands must be executed on the client machine.

Note

"yum update cns-deploy -y" is not required to be executed if OCS 3.10 was deployed via Ansible.

Execute the following steps to stop the Heketi pod to prevent it from accepting any new request for volume creation or volume deletion:
1. Execute the following command to access your project:
```
# oc project <project_name>
```
  For example:
```
# oc project storage-project
```
2. Execute the following command to get the DeploymentConfig:
```
# oc get ds
```
3. Execute the following command to set heketi server to accept requests only from the local-client:
```
# heketi-cli server mode set local-client
```
4. Wait for the ongoing operations to complete and execute the following command to monitor if there are any ongoing operations:
```
# heketi-cli server operations info
```
Backup the Heketi database file
```
# heketi-cli db dump > heketi-db-dump-$(date -I).json
```
Note
The json file created can be used to restore and therefore should be stored in persistent storage of your choice.
Execute the following command to update the heketi client packages. Update the heketi-client package on all the OCP nodes where it is installed. Newer installations may not have the heketi-client rpm installed on any OCP nodes:
```
# yum update heketi-client -y
```
Execute the following command to get the current HEKETI_ADMIN_KEY.
The OCS admin can choose to set any phrase for user key as long as it is not used by their infrastructure. It is not used by any of the OCS default installed resources.
```
# oc get secret heketi-registry-admin-secret -o jsonpath='{.data.key}'|base64 -d;echo
```
If the HEKETI_USER_KEY was set previously, you can obtain it by using the following command:
```
# oc describe pod <heketi-pod>
```

Execute the following step to edit the template:

If the existing template has IMAGE_NAME, then edit the template to change the HEKETI_USER_KEY, HEKETI_ADMIN_KEY, HEKETI_ROUTE, IMAGE_NAME, CLUSTER_NAME and HEKETI_LVM_WRAPPER as shown in the example below.

# oc edit template heketi
parameters:
- description: Set secret for those creating volumes as type user
  displayName: Heketi User Secret
  name: HEKETI_USER_KEY
  value: <heketiuserkey>
- description: Set secret for administration of the Heketi service as user admin
  displayName: Heketi Administrator Secret
  name: HEKETI_ADMIN_KEY
  value: <adminkey>
- description: Set the executor type, kubernetes or ssh
  displayName: heketi executor type
  name: HEKETI_EXECUTOR
  value: kubernetes
- description: Set the hostname for the route URL
  displayName: heketi route name
  name: HEKETI_ROUTE
  value: heketi-registry
- displayName: heketi container image name
  name: IMAGE_NAME
  required: true
  value: registry.redhat.io/rhgs3/rhgs-volmanager-rhel7:v3.11.8
- description: A unique name to identify this heketi service, useful for running
  multiple heketi instances
  displayName: GlusterFS cluster name
  name: CLUSTER_NAME
  value: registry
- description: Heketi can use a wrapper to execute LVM commands, i.e. run commands in the host namespace instead of in the Gluster container
  name: HEKETI_LVM_WRAPPER
  displayName: Wrapper for executing LVM commands
  value: /usr/sbin/exec-on-host

# oc edit template heketi
parameters:
- description: Set secret for those creating volumes as type user
  displayName: Heketi User Secret
  name: HEKETI_USER_KEY
  value: <heketiuserkey>
- description: Set secret for administration of the Heketi service as user admin
  displayName: Heketi Administrator Secret
  name: HEKETI_ADMIN_KEY
  value: <adminkey>
- description: Set the executor type, kubernetes or ssh
  displayName: heketi executor type
  name: HEKETI_EXECUTOR
  value: kubernetes
- description: Set the hostname for the route URL
  displayName: heketi route name
  name: HEKETI_ROUTE
  value: heketi-registry
- displayName: heketi container image name
  name: IMAGE_NAME
  required: true
  value: registry.redhat.io/rhgs3/rhgs-volmanager-rhel7
- displayName: heketi container image version
  name: IMAGE_VERSION
  required: true
  value: v3.11.8
- description: A unique name to identify this heketi service, useful for running multiple heketi instances
  displayName: GlusterFS-registry cluster name
  name: CLUSTER_NAME
  value: registry
- description: Heketi can use a wrapper to execute LVM commands, i.e. run commands in the host namespace instead of in the Gluster container
  name: HEKETI_LVM_WRAPPER
  displayName: Wrapper for executing LVM commands
  value: /usr/sbin/exec-on-host

Note

If a cluster has more than 1000 volumes refer to How to change the default PVS limit in Openshift Container Storage and add the required parameters before proceeding with the upgrade.

Execute the following command to delete the deployment configuration, service, and route for heketi:
```
# oc delete deploymentconfig,service,route heketi-registry
```
Execute the following command to deploy the Heketi service, route, and deployment configuration which will be used to create persistent volumes for OpenShift:
```
# oc process heketi | oc create -f -

service "heketi-registry" created
route "heketi-registry" created
deploymentconfig-registry "heketi" created
```
Note
It is recommended that the heketidbstorage volume be tuned for db workloads. Newly installed Openshift Container Storage deployments tune the heketidbstorage volume automatically. For older deployments, follow the KCS article Planning to run containerized DB or nosql workloads on Openshift Container Storage? and perform the volume set operation for the volume heketidbstorage.

Execute the following command to verify that the containers are running:

# oc get pods

For example:

# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d

6.2.4.2. Upgrading the Red Hat Gluster Storage Registry Pods

The following commands must be executed on the client machine.

Following are the steps for updating a DaemonSet for glusterfs:

Execute the following steps to stop the Heketi pod to prevent it from accepting any new request for volume creation or volume deletion:
1. Execute the following command to access your project:
```
# oc project <project_name>
```
  For example:
```
# oc project storage-project
```
2. Execute the following command to get the DeploymentConfig:
```
# oc get dc
```
3. Execute the following command to set heketi server to accept requests only from the local-client:
```
# heketi-cli server mode set local-client
```
4. Wait for the ongoing operations to complete and execute the following command to monitor if there are any ongoing operations:
```
# heketi-cli server operations info
```
5. Execute the following command to reduce the replica count from 1 to 0. This brings down the Heketi pod:
```
# oc scale dc <heketi_dc> --replicas=0
```
6. Execute the following command to verify that the heketi pod is no longer present:
```
# oc get pods
```
Execute the following command to find the DaemonSet name for gluster
```
# oc get ds
```
Execute the following command to delete the DaemonSet:
```
# oc delete ds <ds-name> --cascade=false
```
Using --cascade=false option while deleting the old DaemonSet does not delete the glusterfs_registry pods but deletes only the DaemonSet. After deleting the old DaemonSet, you must load the new one. When you manually delete the old pods, the new pods which are created will have the configurations of the new DaemonSet.
For example,
```
# oc delete ds glusterfs-registry --cascade=false
daemonset "glusterfs-registry" deleted
```

Execute the following commands to verify all the old pods are up:

# oc get pods

For example,

# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d

Execute the following command to delete the old glusterfs template.
```
 # oc delete templates glusterfs
```

Execute the following command to register new glusterfs template.

# oc create -f /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/glusterfs-template.yml
template "glusterfs" created

Execute the following command to edit the old glusterfs template.

If the template has IMAGE_NAME, then update the glusterfs template as following. For example:

# oc edit template glusterfs

- description: Labels which define the daemonset node selector. Must contain at least
    one label of the format \'glusterfs=<CLUSTER_NAME>-host\'
  displayName: Daemonset Node Labels
  name: NODE_LABELS
  value: '{ "glusterfs": "registry-host" }'
- displayName: GlusterFS container image name
  name: IMAGE_NAME
  required: true
  value: registry.redhat.io/rhgs3/rhgs-server-rhel7:v3.11.8
- description: A unique name to identify which heketi service manages this cluster,
    useful for running multiple heketi instances
  displayName: GlusterFS cluster name
  name: CLUSTER_NAME
  value: registry

If the template has IMAGE_NAME and IMAGE_VERSION as two separate parameters, then update the glusterfs template as following. For example:

# oc edit template glusterfs
- description: Labels which define the daemonset node selector. Must contain at least one label of the format \'glusterfs=<CLUSTER_NAME>-host\'
displayName: Daemonset Node Labels
name: NODE_LABELS
value: '{ "glusterfs": "registry-host" }'
- displayName: GlusterFS container image name
name: IMAGE_NAME
required: true
value: registry.redhat.io/rhgs3/rhgs-server-rhel7
- description: A unique name to identify which heketi service manages this cluster,
useful for running multiple heketi instances
- displayName: GlusterFS container image version
name: IMAGE_VERSION
required: true
value: v3.11.8
- displayName: GlusterFS cluster name
name: CLUSTER_NAME
value: registry

Note

Ensure that the CLUSTER_NAME variable is set to the correct value
If a cluster has more than 1000 volumes refer to How to change the default PVS limit in Openshift Container Storage and add the required parameters before proceeding with the upgrade.

Label all the OpenShift Container Platform nodes that has the Red Hat Gluster Storage pods:
1. Check if the nodes are labelled with the appropriate label by using the following command:
```
# oc get nodes -l glusterfs=registry-host
```

name: kernel-modules mountPath: "/usr/lib/modules" readOnly: true
name: host-rootfs mountPath: "/rootfs"

name: kernel-modules hostPath: path: "/usr/lib/modules"
name: host-rootfs hostPath: path: "/"
displayName: heketi container image name name: IMAGE_NAME required: true value: registry.redhat.io/rhgs3/rhgs-volmanager-rhel7
displayName: heketi container image version name: IMAGE_VERSION required: true value: v3.11.8
1. Execute the following commands to create the gluster DaemonSet:
```
# oc process glusterfs | oc create -f -
```
  For example,
```
# oc process glusterfs | oc create -f -
Deamonset “glusterfs-registry” created
```
2. Execute the following command to identify the old glusterfs_registry pods that needs to be deleted:
```
# oc get pods
```
  For example,
```
# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d
```
3. Execute the following command and ensure that the bricks are not more than 90% full:
```
# df -kh | grep -v ^Filesystem | awk '{if(int($5)>90) print $0}'
```
  Note
  If the bricks are close to 100% utilization, then the Logical Volume Manager(LVM) activation for these bricks may take a long time or can get stuck once the pod or node is rebooted. It is advised to bring down the utilization of that brick or expand the physical volume(PV) that is using the logical volume(LV).
  Note
  The df command is not applicable to bricks that belong to Block Hosting Volume(BHV). On a BHV , the used size of the bricks generated by the df command is the added size of the blockvolumes of that Gluster volume, it is not the size of the data that resides in the blockvolumes. For more information refer to How To Identify Block Volumes and Block Hosting Volumes in Openshift Container Storage.
4. Execute the following command to delete the old glusterfs-registry pods. glusterfs-registry pods should follow rolling upgrade. Hence, you must ensure that the new pod is running before deleting the next old glusterfs-registry pods. We support OnDelete Strategy DaemonSet update strategy. With OnDelete Strategy update strategy, after you update a DaemonSet template, new DaemonSet pods will only be created when you manually delete old DaemonSet pods.
  1. To delete the old glusterfs-registry pods, execute the following command:
    # oc delete pod <gluster_pod>
    For example,
    # oc delete pod glusterfs-registry-4cpcc pod “glusterfs-registry-4cpcc” deleted
    Note
    Before deleting the next pod, self heal check has to be made:
    Run the following command to access shell on glusterfs-registry pods:
    # oc rsh <gluster_pod_name>
    Run the following command to check the self-heal status of all the volumes: :
    # for eachVolume in $(gluster volume list); do gluster volume heal $eachVolume info ; done | grep "Number of entries: [^0]$"
  2. The delete pod command will terminate the old pod and create a new pod. Run # oc get pods -w and check the Age of the pod and READY status should be 1/1. The following is the example output showing the status progression from termination to creation of the pod.
    # oc get pods -w NAME READY STATUS RESTARTS AGE glusterfs-registry-4cpcc 1/1 Terminating 0 3d …
5. Execute the following command to verify that the pods are running:
```
# oc get pods
```
  For example,
```
# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d
```
6. Execute the following commands to verify if you have upgraded the pod to the latest version:
```
# oc rsh <gluster_registry_pod_name> glusterd --version
```
  For example:
```
# oc rsh glusterfs-registry-abmqa glusterd --version
glusterfs 6.0
```
```
# rpm -qa|grep gluster
```
7. Check the Red Hat Gluster Storage op-version by executing the following command on one of the glusterfs-registry pods.
```
# gluster vol get all cluster.op-version
```
8. After you upgrade the Gluster pods, ensure that you set Heketi back to operational mode:
  - Scale up the DC (Deployment Configuration).
    # oc scale dc <heketi_dc> --replicas=1
9. Set the cluster.op-version to 70200 on any one of the pods:
  Note
  Ensure all the glusterfs-registry pods are updated before changing the cluster.op-version.
```
# gluster volume set all cluster.op-version 70200
```
10. Execute the following steps to enable server.tcp-user-timeout on all volumes.
  Note
  The "server.tcp-user-timeout" option specifies the maximum amount of the time (in seconds) the transmitted data from the application can remain unacknowledged from the brick.
  It is used to detect force disconnections and dead connections (if a node dies unexpectedly, a firewall is activated, etc.,) early and make it possible for applications to reduce the overall failover time.
  1. List the glusterfs pod using the following command:
    # oc get pods
    For example:
    # oc get pods NAME READY STATUS RESTARTS AGE glusterblock-storage-provisioner-dc-1-ffgs5 1/1 Running 0 3m glusterfs-storage-5thpc 1/1 Running 0 9d glusterfs-storage-hfttr 1/1 Running 0 9d glusterfs-storage-n8rg5 1/1 Running 0 9d heketi-storage-4-9fnvz 2/2 Running 0 8d
  2. Remote shell into one of the glusterfs-registry pods. For example:
    # oc rsh glusterfs-registry-g6vd9
  3. Execute the following command:
    # for eachVolume in `gluster volume list`; do echo $eachVolume; gluster volume set $eachVolume server.tcp-user-timeout 42 ; done
    For example:
    # for eachVolume in `gluster volume list`; do echo $eachVolume; gluster volume set $eachVolume server.tcp-user-timeout 42 ; done volume1 volume set: success volume2 volume set: success
11. If a gluster-block-registry-provisoner-pod already exists then delete it by executing the following commands:
```
# oc delete dc <gluster-block-registry-dc>
```
  For example:
```
# oc delete dc glusterblock-registry-provisioner-dc
```
12. Execute the following command to delete the old glusterblock provisioner template.
```
 # oc delete templates glusterblock-provisioner
```
13. Create a glusterblock provisioner template. For example:
```
# oc create -f /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/glusterblock-provisioner.yml
template.template.openshift.io/glusterblock-provisioner created
```
14. Depending on the OCP version, edit the glusterblock-provisioner template to change the IMAGE_NAME and NAMESPACE.
```
# oc edit template glusterblock-provisioner
- displayName: glusterblock provisioner container image name
  name: IMAGE_NAME
  required: true
  value: registry.redhat.io/rhgs3/rhgs-gluster-block-prov-rhel7:v3.11.8
- description: The namespace in which these resources are being created
  displayName: glusterblock provisioner namespace
  name: NAMESPACE
  required: true
  value: glusterfs-registry
- description: A unique name to identify which heketi service manages this cluster, useful for running multiple heketi instances
  displayName: GlusterFS cluster name
  name: CLUSTER_NAME
  value: registry
```
  - If the template has IMAGE_NAME and IMAGE_VERSION as two separate parameters, then update the glusterblock-provisioner template as following.
    For example:
    # oc edit template glusterblock-provisioner - displayName: glusterblock provisioner container image name name: IMAGE_NAME required: true value: registry.redhat.io/rhgs3/rhgs-gluster-block-prov-rhel7 - displayName: glusterblock provisioner container image version name: IMAGE_VERSION required: true value: v3.11.8 - description: The namespace in which these resources are being created displayName: glusterblock provisioner namespace name: NAMESPACE required: true value: glusterfs-registry - description: A unique name to identify which heketi service manages this cluster, useful for running multiple heketi instances displayName: GlusterFS cluster name name: CLUSTER_NAME value: registry
15. Delete the following resources from the old pod
```
# oc delete clusterroles.authorization.openshift.io glusterblock-provisioner-runner
# oc delete serviceaccounts glusterblock-registry-provisioner
# oc delete clusterrolebindings.authorization.openshift.io glusterblock-registry-provisioner
```
16. Before running oc process determine the correct provisioner name. If there are more than one gluster block provisioner running in your cluster the names must differ from all other provisioners.
  For example,
  - If there are 2 or more provisioners the name should be gluster.org/glusterblock-<namespace> where, namespace is replaced by the namespace that the provisioner is deployed in.
  - If there is only one provisioner, installed prior to 3.11.8, gluster.org/glusterblock is sufficent. If the name currently in use already has a unique namespace suffix, reuse the existing name.
17. After editing the template, execute the following command to create the deployment configuration:
```
# oc process glusterblock-provisioner -o yaml | oc create -f -
```
  For example:
```
# oc process glusterblock-provisioner -o yaml | oc create -f -
clusterrole.authorization.openshift.io/glusterblock-provisioner-runner created
serviceaccount/glusterblock-registry-provisioner created
clusterrolebinding.authorization.openshift.io/glusterblock-registry-provisioner created
deploymentconfig.apps.openshift.io/glusterblock-registry-provisioner-dc created
```
18. Brick multiplexing is a feature that allows adding multiple bricks into one process. This reduces resource consumption and allows us to run more bricks than before with the same memory consumption. It is enabled by default from Container-Native Storage 3.6 onward. During an upgrade from Container-Native Storage 3.10 to Red Hat Openshift Container Storage 3.11, to turn brick multiplexing on, execute the following commands:
  1. To exec into the Gluster pod, execute the following command and rsh into any of the gluster pods:
    # oc rsh <gluster_pod_name>
  2. Verify the brick multiplex status:
    # gluster v get all all
  3. If it is disabled, then execute the following command to enable brick multiplexing:
    Note
    Ensure that all volumes are in a stop state or no bricks are running while brick multiplexing is enabled.
    # gluster volume set all cluster.brick-multiplex on
    For example:
    # oc rsh glusterfs-registry-g6vd9 sh-4.2# gluster volume set all cluster.brick-multiplex on Brick-multiplexing is supported only for container workloads (Independent or Converged mode). Also it is advised to make sure that either all volumes are in stopped state or no bricks are running before this option is modified.Do you still want to continue? (y/n) y volume set: success
  4. List all the volumes in the trusted storage pool. This step is only required if the volume set operation is performed:
    For example:
    # gluster volume list heketidbstorage vol_194049d2565d2a4ad78ef0483e04711e ... ...
    Restart all the volumes. This step is only required if the volume set operation is performed along with the previous step:
    # gluster vol stop <VOLNAME> # gluster vol start <VOLNAME>
19. Support for S3 compatible Object Store in Red Hat Openshift Container Storage is under technology preview. To enable S3 compatible object store, see https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html/operations_guide/s3_object_store.
  Note
  After upgrading the glusterfs registry pods, proceed with the steps listed in ] to bring back your heketi pod and then proceed with the steps listed in xref:chap-upgrade_client_common[ to upgrade the client on Red Hat Openshift Container Platform Nodes.
20. All storage classes that use gluster block volume provisioning must match exactly to one of the provisioner names in the cluster. To check the list of storage classes that refer to a block provisioner, in a given namespace, run the following command:
```
# oc get sc -o custom-columns=NAME:.metadata.name,PROV:.provisioner,RSNS:.parameters.restsecretnamespace | grep 'gluster.org/glusterblock' | grep <namespace>
```
  Example:
```
# oc get sc -o custom-columns=NAME:.metadata.name,PROV:.provisioner,RSNS:.parameters.restsecretnamespace | grep 'gluster.org/glusterblock' | grep infra-storage
  glusterfs-registry-block   gluster.org/glusterblock               infra-storage
```
  Check each storage class provisioner name, if it does not match the block provisioner name configured for that namespace it must be updated. If the block provisioner name already matches the configured provisioner name, nothing else needs to be done. Use the list generated above and include all storage class names where the provionser name must be updated.
  For every storage class in this list do the following:
```
# oc get sc  -o yaml <storageclass>  > storageclass-to-edit.yaml
# oc delete sc  <storageclass>
# sed 's,gluster.org/glusterblock$,gluster.org/glusterblock-<namespace>,' storageclass-to-edit.yaml | oc create -f -
```
  Example:
```
# oc get sc -o yaml glusterfs-registry-block > storageclass-to-edit.yaml
# oc delete sc glusterfs-registry-block
storageclass.storage.k8s.io "glusterfs-registry-block" deleted
# sed 's,gluster.org/glusterblock$,gluster.org/glusterblock-infra-storage,' storageclass-to-edit.yaml | oc create -f -
storageclass.storage.k8s.io/glusterfs-registry-block created
```

6.3. Starting the Heketi Pods

Execute the following commands on the client machine for both glusterfs and registry namespace.

Execute the following command to navigate to the project where the Heketi pods are running:
```
# oc project <project_name>
```
For example for glusterfs namespace:
```
# oc project glusterfs
```
For example for registry namespace:
```
# oc project glusterfs-registry
```

Execute the following command to get the DeploymentConfig:

# oc get dc

For example, on a glusterfs-registry project:

# oc get dc
NAME                                  REVISION   DESIRED   CURRENT   TRIGGERED BY
glusterblock-storage-provisioner-dc   1          1         0         config
heketi-storage                        4          1         1         config

For example, on a glusterfs project:

# oc get dc
NAME                                  REVISION   DESIRED   CURRENT   TRIGGERED BY
glusterblock-storage-provisioner-dc   1          1         0         config
heketi-storage                        4          1         1         config

Execute the following command to increase the replica count from 0 to 1. This brings back the Heketi pod:
```
# oc scale dc <heketi_dc> --replicas=1
```

Execute the following command to verify that the heketi pod is present in both glusterfs and glusterfs-registry namespace:

# oc get pods

For example for glusterfs:

# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d

For example for registry pods:

# oc get pods
NAME                                          READY     STATUS    RESTARTS   AGE
glusterblock-storage-provisioner-dc-1-ffgs5   1/1       Running   0          3m
glusterfs-storage-5thpc                       1/1       Running   0          9d
glusterfs-storage-hfttr                       1/1       Running   0          9d
glusterfs-storage-n8rg5                       1/1       Running   0          9d
heketi-storage-4-9fnvz                        2/2       Running   0          8d

6.4. Upgrading the client on Red Hat OpenShift Container Platform nodes

Execute the following commands on each of the nodes:

To drain the pod, execute the following command on the master node (or any node with cluster-admin access):
```
# oc adm drain <node_name> --ignore-daemonsets
```
To check if all the pods are drained, execute the following command on the master node (or any node with cluster-admin access) :
```
# oc get pods --all-namespaces --field-selector=spec.nodeName=<node_name>
```
Execute the following command to upgrade the client node to the latest glusterfs-fuse version:
```
# yum update glusterfs-fuse
```
To enable node for pod scheduling execute the following command on the master node (or any node with cluster-admin access):
```
# oc adm manage-node --schedulable=true <node_name>
```

Create and add the following content to the multipath.conf file:

Note

The multipath.conf file does not require any change as the change was implemented during a previous upgrade.

# cat >> /etc/multipath.conf <<EOF
# LIO iSCSI
devices {
  device {
    vendor "LIO-ORG"
    user_friendly_names "yes" # names like mpatha
    path_grouping_policy "failover" # one path per group
    hardware_handler "1 alua"
    path_selector "round-robin 0"
    failback immediate
    path_checker "tur"
    prio "alua"
    no_path_retry 120
    rr_weight "uniform"
  }
}
EOF

Execute the following commands to start multipath daemon and [re]load the multipath configuration:
```
# systemctl start multipathd
```
```
# systemctl reload multipathd
```

Select Your Language

Chapter 6. Upgrading your Red Hat Openshift Container Storage in Converged Mode

6.1. Upgrading the pods in the glusterfs group

6.1.1. Prerequisites

6.1.2. Restoring original label values for /dev/log

6.1.3. Upgrading if existing version deployed by using cns-deploy

6.1.3.1. Upgrading cns-deploy and Heketi Server

6.1.3.2. Upgrading the Red Hat Gluster Storage Pods

6.1.4. Upgrading if existing version deployed by using Ansible

6.1.4.1. Upgrading Heketi Server

6.1.4.2. Upgrading the Red Hat Gluster Storage Pods

6.2. Upgrading the pods in the glusterfs registry group

6.2.1. Prerequisites

6.2.2. Restoring original label values for /dev/log

6.2.3. Upgrading if existing version deployed by using cns-deploy

6.2.3.1. Upgrading cns-deploy and Heketi Server

6.2.3.2. Upgrading the Red Hat Gluster Storage Registry Pods

6.2.4. Upgrading if existing version deployed by using Ansible

6.2.4.1. Upgrading Heketi Server

6.2.4.2. Upgrading the Red Hat Gluster Storage Registry Pods

6.3. Starting the Heketi Pods

6.4. Upgrading the client on Red Hat OpenShift Container Platform nodes

Quick Links

Help

Site Info

Related Sites

About

Red Hat legal and privacy links

Red Hat legal and privacy links

Language and Page Formatting Options

Chapter 6. Upgrading your Red Hat Openshift Container Storage in Converged Mode

6.1. Upgrading the pods in the glusterfs group

6.1.1. Prerequisites

6.1.2. Restoring original label values for /dev/log

6.1.3. Upgrading if existing version deployed by using cns-deploy

6.1.3.1. Upgrading cns-deploy and Heketi Server

6.1.3.2. Upgrading the Red Hat Gluster Storage Pods

6.1.4. Upgrading if existing version deployed by using Ansible

6.1.4.1. Upgrading Heketi Server

6.1.4.2. Upgrading the Red Hat Gluster Storage Pods

6.2. Upgrading the pods in the glusterfs registry group

6.2.1. Prerequisites

6.2.2. Restoring original label values for /dev/log

6.2.3. Upgrading if existing version deployed by using cns-deploy

6.2.3.1. Upgrading cns-deploy and Heketi Server

6.2.3.2. Upgrading the Red Hat Gluster Storage Registry Pods

6.2.4. Upgrading if existing version deployed by using Ansible

6.2.4.1. Upgrading Heketi Server

6.2.4.2. Upgrading the Red Hat Gluster Storage Registry Pods

6.3. Starting the Heketi Pods

6.4. Upgrading the client on Red Hat OpenShift Container Platform nodes

Quick Links

Help

Site Info

Related Sites

Systems Status

About

Red Hat legal and privacy links

Red Hat legal and privacy links