Chapter 6. Upgrading your Red Hat Openshift Container Storage in Converged Mode

This chapter describes the procedure to upgrade your environment from Container Storage in Converged Mode 3.10 to Red Hat Openshift Container Storage in Converged Mode 3.11.

Note
  • New registry name registry.redhat.io is used throughout in this Guide. However, if you have not migrated to the new registry yet then replace all occurrences of registry.redhat.io with registry.access.redhat.com where ever applicable.
  • Follow the same upgrade procedure to upgrade your environment from Red Hat Openshift Container Storage in Converged Mode 3.11.0 and above to Red Hat Openshift Container Storage in Converged Mode 3.11.5. Ensure that the correct image and version numbers are configured before you start the upgrade process.
  • The valid images for Red Hat Openshift Container Storage 3.11.5 are:

    • registry.redhat.io/rhgs3/rhgs-server-rhel7:v3.11.6
    • registry.redhat.io/rhgs3/rhgs-volmanager-rhel7:v3.11.6
    • registry.redhat.io/rhgs3/rhgs-gluster-block-prov-rhel7:v3.11.6
    • registry.redhat.io/rhgs3/rhgs-s3-server-rhel7:v3.11.6

6.1. Upgrading the pods in the glusterfs group

The following sections provide steps to upgrade your Glusterfs pods.

6.1.1. Prerequisites

Ensure the following prerequisites are met:

Note

For deployments using cns-deploy tool, the templates are available in the following location:

  • gluster template - /usr/share/heketi/templates/glusterfs-template.yaml
  • heketi template - /usr/share/heketi/templates/heketi-template.yaml
  • glusterblock-provisioner template - /usr/share/heketi/templates/glusterblock-provisioner.yaml

For deployments using ansible playbook the templates are available in the following location:

  • gluster template - /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/glusterfs-template.yml
  • heketi template - /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/heketi-template.yml
  • glusterblock-provisioner template - /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/glusterblock-provisioner.yml

6.1.2. Restoring original label values for /dev/log

Note

Follow this procedure only if you are upgrading your environment from Red Hat Container Native Storage 3.9 to Red Hat Openshift Container Storage 3.11.5.

Skip this procedure if you are upgrading your environment from Red Hat Openshift Container Storage 3.10 and above to Red Hat Openshift Container Storage 3.11.5.

To restore the original selinux label, execute the following commands:

  1. Create a directory and soft links on all nodes that run gluster pods:

    # mkdir /srv/<directory_name>
    # cd /srv/<directory_name>/   # same dir as above
    # ln -sf /dev/null systemd-tmpfiles-setup-dev.service
    # ln -sf /dev/null systemd-journald.service
    # ln -sf /dev/null systemd-journald.socket
  2. Edit the daemonset that creates the glusterfs pods on the node which has oc client:

    # oc edit daemonset <daemonset_name>

    Under volumeMounts section add a mapping for the volume:

    - mountPath: /usr/lib/systemd/system/systemd-journald.service
      name: systemd-journald-service
    - mountPath: /usr/lib/systemd/system/systemd-journald.socket
      name: systemd-journald-socket
    - mountPath: /usr/lib/systemd/system/systemd-tmpfiles-setup-dev.service
    name: systemd-tmpfiles-setup-dev-service

    Under volumes section add a new host path for each service listed:

    Note

    The path mentioned in here should be the same as mentioned in Step 1.

    - hostPath:
       path: /srv/<directory_name>/systemd-journald.socket
       type: ""
      name: systemd-journald-socket
    - hostPath:
       path: /srv/<directory_name>/systemd-journald.service
       type: ""
      name: systemd-journald-service
    - hostPath:
       path: /srv/<directory_name>/systemd-tmpfiles-setup-dev.service
       type: ""
    name: systemd-tmpfiles-setup-dev-service
  3. Run the following command on all nodes that run gluster pods. This will reset the label:

    # restorecon /dev/log
  4. Execute the following command to check the status of self heal for all volumes:

    # oc rsh <gluster_pod_name>
    # for each_volume in `gluster volume list`; do gluster volume heal $each_volume info ; done  | grep  "Number of entries: [^0]$"

    Wait for self-heal to complete.

  5. Execute the following command and ensure that the bricks are not more than 90% full:

    # df -kh | grep -v ^Filesystem | awk '{if(int($5)>90) print $0}'
    Note

    If the bricks are close to 100% utilization, then the Logical Volume Manager(LVM) activation for these bricks may take a long time or can get stuck once the pod or node is rebooted. It is advised to bring down the utilization of that brick or expand the physical volume(PV) that is using the logical volume(LV).

  6. Execute the following command on any one of the gluster pods to set the maximum number of bricks (250) that can run on a single instance of glusterfsd process:

    # gluster volume set all cluster.max-bricks-per-process 250
    1. Execute the following command on any one of the gluster pods to ensure that the option is set correctly:

      # gluster volume get all cluster.max-bricks-per-process

      For example:

      # gluster volume get all cluster.max-bricks-per-process
      cluster.max-bricks-per-process 250
  7. Execute the following command on the node which has oc client to delete the gluster pod:

    # oc delete pod <gluster_pod_name>
  8. To verify if the pod is ready, execute the following command:

    # oc get pods -l glusterfs=storage-pod
  9. Login to the node hosting the pod and check the selinux label of /dev/log

    # ls -lZ /dev/log

    The output should show devlog_t label

    For example:

    #  ls -lZ /dev/log
    srw-rw-rw-. root root system_u:object_r:devlog_t:s0    /dev/log

    Exit the node.

  10. In the gluster pod, check if the label value is devlog_t:

    # oc rsh <gluster_pod_name>
    # ls -lZ /dev/log

    For example:

    #  ls -lZ /dev/log
    srw-rw-rw-. root root system_u:object_r:devlog_t:s0    /dev/log
  11. Perform steps 4 to 9 for other pods.

6.1.3. Upgrading if existing version deployed by using cns-deploy

6.1.3.1. Upgrading cns-deploy and Heketi Server

The following commands must be executed on the client machine.

  1. Execute the following command to update the heketi client and cns-deploy packages:

    # yum update cns-deploy -y
    # yum update heketi-client -y
  2. Backup the Heketi database file

    # heketi-cli db dump > heketi-db-dump-$(date -I).json
    • Execute the following command to get the current HEKETI_ADMIN_KEY.

      The OCS admin can choose to set any phrase for user key as long as it is not used by their infrastructure. It is not used by any of the OCS default installed resources.

      oc get secret <heketi-admin-secret> -o jsonpath='{.data.key}'|base64 -d;echo
  3. Execute the following command to delete the heketi template.

    # oc delete templates heketi
  4. Execute the following command to install the heketi template.

    oc create -f /usr/share/heketi/templates/heketi-template.yaml
    template "heketi" created
  5. Execute the following command to grant the heketi Service Account the necessary privileges.

    # oc policy add-role-to-user edit system:serviceaccount:<project_name>:heketi-service-account
    # oc adm policy add-scc-to-user privileged -z heketi-service-account

    For example,

    # oc policy add-role-to-user edit system:serviceaccount:storage-project:heketi-service-account
    # oc adm policy add-scc-to-user privileged -z heketi-service-account
  6. Execute the following command to generate a new heketi configuration file.

    # sed -e "s/\${HEKETI_EXECUTOR}/kubernetes/" -e "s#\${HEKETI_FSTAB}#/var/lib/heketi/fstab#" -e "s/\${SSH_PORT}/22/" -e "s/\${SSH_USER}/root/" -e "s/\${SSH_SUDO}/false/" -e "s/\${BLOCK_HOST_CREATE}/true/" -e "s/\${BLOCK_HOST_SIZE}/500/" "/usr/share/heketi/templates/heketi.json.template" > heketi.json
    • The BLOCK_HOST_SIZE parameter controls the size (in GB) of the automatically created Red Hat Gluster Storage volumes hosting the gluster-block volumes (For more information, see https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html-single/operations_guide/index#Block_Storage). This default configuration will dynamically create block-hosting volumes of 500GB in size as more space is required.
    • Alternatively, copy the file /usr/share/heketi/templates/heketi.json.template to heketi.json in the current directory and edit the new file directly, replacing each "${VARIABLE}" string with the required parameter.

      Note

      JSON formatting is strictly required (e.g. no trailing spaces, booleans in all lowercase).

  7. Execute the following command to create a secret to hold the configuration file.

    # oc create secret generic <heketi-config-secret> --from-file=heketi.json
    Note

    If the heketi-config-secret file already exists, then delete the file and run the following command.

  8. Execute the following command to delete the deployment configuration, service, and route for heketi:

    # oc delete deploymentconfig,service,route heketi
    Note

    The names of these parameters can be referenced from output of the following command:

    # oc get all | grep heketi
  9. Execute the following command to edit the heketi template. Edit the HEKETI_USER_KEY and HEKETI_ADMIN_KEY parameters.

    # oc edit template heketi
    parameters:
    - description: Set secret for those creating volumes as type user
      displayName: Heketi User Secret
      name: HEKETI_USER_KEY
      value: <heketiuserkey>
    - description: Set secret for administration of the Heketi service as user admin
      displayName: Heketi Administrator Secret
      name: HEKETI_ADMIN_KEY
      value: <adminkey>
    - description: Set the executor type, kubernetes or ssh
      displayName: heketi executor type
      name: HEKETI_EXECUTOR
      value: kubernetes
    - description: Set the hostname for the route URL
      displayName: heketi route name
      name: HEKETI_ROUTE
      value: heketi-storage
    - displayName: heketi container image name
      name: IMAGE_NAME
      required: true
      value: registry.redhat.io/rhgs3/rhgs-volmanager-rhel7
    - displayName: heketi container image version
      name: IMAGE_VERSION
      required: true
      value: v3.11.6
    - description: A unique name to identify this heketi service, useful for running
        multiple heketi instances
      displayName: GlusterFS cluster name
      name: CLUSTER_NAME
      value: storage
    Note

    If a cluster has more than 1000 volumes refer to How to change the default PVS limit in Openshift Container Storage and add the required parameters before proceeding with the upgrade.

  10. Execute the following command to deploy the Heketi service, route, and deployment configuration which will be used to create persistent volumes for OpenShift:

    # oc process heketi | oc create -f -
    
      service "heketi" created
      route "heketi" created
    deploymentconfig "heketi" created
    Note

    It is recommended that the heketidbstorage volume be tuned for db workloads. Newly installed Openshift Container Storage deployments tune the heketidbstorage volume automatically. For older deployments, follow the KCS article Planning to run containerized DB or nosql workloads on Openshift Container Storage? and perform the volume set operation for the volume heketidbstorage.

  11. Execute the following command to verify that the containers are running:

    # oc get pods

    For example:

    # oc get pods
      NAME                             READY     STATUS    RESTARTS   AGE
      glusterfs-0h68l                  1/1       Running   0          3d
      glusterfs-0vcf3                  1/1       Running   0          3d
      glusterfs-gr9gh                  1/1       Running   0          3d
      heketi-1-zpw4d                   1/1       Running   0          3h
      storage-project-router-2-db2wl   1/1       Running   0          4d

6.1.3.2. Upgrading the Red Hat Gluster Storage Pods

The following commands must be executed on the client machine.

Following are the steps for updating a DaemonSet for glusterfs:

  1. Execute the following steps to stop the Heketi pod to prevent it from accepting any new request for volume creation or volume deletion:

    1. Execute the following command to access your project:

      # oc project <project_name>

      For example:

      # oc project storage-project
    2. Execute the following command to get the DeploymentConfig:

      # oc get ds
    3. Execute the following command to set heketi server to accept requests only from the local-client:

      # heketi-cli server mode set local-client
    4. Wait for the ongoing operations to complete and execute the following command to monitor if there are any ongoing operations:

      # heketi-cli server operations info
    5. Execute the following command to reduce the replica count from 1 to 0. This brings down the Heketi pod:

      # oc scale dc <heketi_dc> --replicas=0
    6. Execute the following command to verify that the heketi pod is no longer present:

      # oc get pods
  2. Execute the following command to find the DaemonSet name for gluster

    # oc get ds
  3. Execute the following command to delete the DaemonSet:

    # oc delete ds <ds-name> --cascade=false

    Using --cascade=false option while deleting the old DaemonSet does not delete the gluster pods but deletes only the DaemonSet. After deleting the old DaemonSet, you must load the new one. When you manually delete the old pods, the new pods which are created will have the configurations of the new DaemonSet.

    For example,

    # oc delete ds glusterfs --cascade=false
    daemonset "glusterfs" deleted
  4. Execute the following commands to verify all the old pods are up:

    # oc get pods

    For example,

    # oc get pods
      NAME                             READY     STATUS    RESTARTS   AGE
      glusterfs-0h68l                  1/1       Running   0          3d
      glusterfs-0vcf3                  1/1       Running   0          3d
      glusterfs-gr9gh                  1/1       Running   0          3d
      storage-project-router-2-db2wl   1/1       Running   0          4d
  5. Execute the following command to delete the old glusterfs template.

    # oc delete templates glusterfs

    For example,

    # oc delete templates glusterfs
    template “glusterfs” deleted
  6. Execute the following command to register new glusterfs template.

    # oc create -f /usr/share/heketi/templates/glusterfs-template.yaml

    For example,

    # oc create -f /usr/share/heketi/templates/glusterfs-template.yaml
      template “glusterfs” created
  7. Label all the OpenShift Container Platform nodes that has the Red Hat Gluster Storage pods:

    1. Check if the nodes are labelled with the appropriate label by using the following command:

      # oc get nodes -l glusterfs=storage-host
  8. Execute the following commands to create the gluster DaemonSet:

    # oc process glusterfs | oc create -f -

    For example,

    # oc process glusterfs | oc create -f -
    Deamonset “glusterfs” created
    Note

    If a cluster has more than 1000 volumes refer to How to change the default PVS limit in Openshift Container Storage and add the required parameters before proceeding with the upgrade.

  9. Execute the following command to identify the old gluster pods that needs to be deleted:

    # oc get pods

    For example,

    # oc get pods
      NAME                             READY     STATUS    RESTARTS   AGE
      glusterfs-0h68l                  1/1       Running   0          3d
      glusterfs-0vcf3                  1/1       Running   0          3d
      glusterfs-gr9gh                  1/1       Running   0          3d
      storage-project-router-2-db2wl   1/1       Running   0          4d
  10. Execute the following command and ensure that the bricks are not more than 90% full:

    # df -kh | grep -v ^Filesystem | awk '{if(int($5)>90) print $0}'
    Note

    If the bricks are close to 100% utilization, then the Logical Volume Manager(LVM) activation for these bricks may take a long time or can get stuck once the pod or node is rebooted. It is advised to bring down the utilization of that brick or expand the physical volume(PV) that is using the logical volume(LV).

  11. Execute the following command to delete the old gluster pods. Gluster pods should follow rolling upgrade. Hence, you must ensure that the new pod is running before deleting the next old gluster pod. We support OnDelete Strategy DaemonSet update strategy. With OnDelete Strategy update strategy, after you update a DaemonSet template, new DaemonSet pods will only be created when you manually delete old DaemonSet pods.

    1. To delete the old gluster pods, execute the following command:

      # oc delete pod <gluster_pod>

      For example,

      # oc delete pod glusterfs-0vcf3
      pod  “glusterfs-0vcf3” deleted
      Note

      Before deleting the next pod, self heal check has to be made:

      1. Run the following command to access shell on gluster pod:

        # oc rsh <gluster_pod_name>
      2. Run the following command to check the self-heal status of all the volumes:

        for each_volume in `gluster volume list`;
          do gluster volume heal $each_volume info ;
        done | grep "Number of entries: [^0]$"
    2. The delete pod command will terminate the old pod and create a new pod. Run # oc get pods -w and check the Age of the pod and READY status should be 1/1. The following is the example output showing the status progression from termination to creation of the pod.

      # oc get pods -w
        NAME                             READY     STATUS        RESTARTS   AGE
        glusterfs-0vcf3                  1/1       Terminating   0          3d
      …
  12. Execute the following command to verify that the pods are running:

    # oc get pods

    For example,

    # oc get pods
      NAME                             READY     STATUS    RESTARTS   AGE
      glusterfs-j241c                  1/1       Running   0          4m
      glusterfs-pqfs6                  1/1       Running   0          7m
      glusterfs-wrn6n                  1/1       Running   0          12m
      storage-project-router-2-db2wl   1/1       Running   0          4d
  13. Execute the following command to verify if you have upgraded the pod to the latest version:

    # oc rsh <gluster_pod_name> glusterd --version

    For example:

     # oc rsh glusterfs-4cpcc glusterd --version
    glusterfs 6.0
  14. Check the Red Hat Gluster Storage op-version by executing the following command on one of the gluster pods.

    # gluster vol get all cluster.op-version
  15. After you upgrade the Gluster pods, ensure that you set Heketi back to operational mode:

    • Scale up the DC (Deployment Configuration).

      # oc scale dc <heketi_dc> --replicas=1
  16. Set the cluster.op-version to 70000 on any one of the pods:

    Important

    Ensure all the gluster pods are updated before changing the cluster.op-version.

    # gluster --timeout=3600 volume set all cluster.op-version 70000
    • Execute the following steps to enable server.tcp-user-timeout on all volumes.

      Note

      The "server.tcp-user-timeout" option specifies the maximum amount of the time (in seconds) the transmitted data from the application can remain unacknowledged from the brick.

      It is used to detect force disconnections and dead connections (if a node dies unexpectedly, a firewall is activated, etc.,) early and make it possible for applications to reduce the overall failover time.

      1. List the glusterfs pod using the following command:

        # oc get pods

        For example:

        # oc get pods
          NAME                             READY     STATUS    RESTARTS   AGE
          glusterfs-0h68l                  1/1       Running   0          3d
          glusterfs-0vcf3                  1/1       Running   0          3d
          glusterfs-gr9gh                  1/1       Running   0          3d
        storage-project-router-2-db2wl   1/1       Running   0          4d
      2. Remote shell into one of the glusterfs pods. For example:

        # oc rsh glusterfs-0vcf3
      3. Execute the following command:

        # for eachVolume in `gluster volume list`; do echo $eachVolume; gluster volume set $eachVolume server.tcp-user-timeout 42 ; done

        For example:

        # for eachVolume in `gluster volume list`; do echo $eachVolume; gluster volume set $eachVolume server.tcp-user-timeout 42 ; done
          volume1
          volume set: success
          volume2
        volume set: success
  17. If a gluster-block-provisoner-pod already exists then delete it by executing the following commands:

    # oc delete dc glusterblock-provisioner-dc

    For example:

    # oc delete dc glusterblock-storage-provisioner-dc
  18. Delete the following resources from the old pod:

    # oc delete clusterroles.authorization.openshift.io glusterblock-provisioner-runner
    # oc delete serviceaccounts glusterblock-provisioner
    serviceaccount "glusterblock-provisioner" deleted
    # oc delete clusterrolebindings.authorization.openshift.io glusterblock-provisioner
  19. Execute the following commands to deploy the gluster-block provisioner:

    # sed -e 's/\\\${NAMESPACE}/<NAMESPACE>/' /usr/share/heketi/templates/glusterblock-provisioner.yaml | oc create -f -
    # oc adm policy add-cluster-role-to-user glusterblock-provisioner-runner system:serviceaccount:<NAMESPACE>:glusterblock-provisioner

    For example:

    # sed -e 's/\\\${NAMESPACE}/storage-project/' /usr/share/heketi/templates/glusterblock-provisioner.yaml | oc create -f -
    # oc adm policy add-cluster-role-to-user glusterblock-provisioner-runner system:serviceaccount:storage-project:glusterblock-provisioner
  20. Brick multiplexing is a feature that allows adding multiple bricks into one process. This reduces resource consumption and allows us to run more bricks than before with the same memory consumption. It is enabled by default from Container-Native Storage 3.6 onward. During an upgrade from Container-Native Storage 3.10 to Red Hat Openshift Container Storage 3.11, to turn brick multiplexing on, execute the following commands:

    1. To exec into the Gluster pod, execute the following command and rsh into any of the gluster pods:

      # oc rsh <gluster_pod_name>
    2. Verify the brick multiplex status:

      # gluster v get all all
    3. If it is disabled, then execute the following command to enable brick multiplexing:

      Note

      Ensure that all volumes are in a stop state or no bricks are running while brick multiplexing is enabled.

      # gluster volume set all cluster.brick-multiplex on

      For example:

      # oc rsh glusterfs-770ql
      
        sh-4.2# gluster volume set all cluster.brick-multiplex on
        Brick-multiplexing is supported only for container workloads (Independent or Converged mode). Also it is advised to make sure that either all volumes are in stopped state or no bricks are running before this option is modified.Do you still want to continue? (y/n) y
      volume set: success
    4. List all the volumes in the trusted storage pool. This step is only required if the volume set operation is performed:

      For example:

      # gluster volume list
      
        heketidbstorage
        vol_194049d2565d2a4ad78ef0483e04711e
        ...
        ...

      Restart all the volumes. This step is only required if the volume set operation is performed along with the previous step:

      # gluster vol stop <VOLNAME>
      # gluster vol start <VOLNAME>
  21. Support for S3 compatible Object Store in Red Hat Openshift Container Storage is under technology preview. To enable S3 compatible object store, see https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html/operations_guide/s3_object_store.
Note

6.1.4. Upgrading if existing version deployed by using Ansible

6.1.4.1. Upgrading Heketi Server

The following commands must be executed on the client machine.

  1. Backup the Heketi database file

    # heketi-cli db dump > heketi-db-dump-$(date -I).json
    Note

    The json file created can be used to restore and therefore should be stored in persistent storage of your choice.

  2. Execute the following command to update the heketi client packages. Update the heketi-client package on all the OCP nodes where it is installed. Newer installations may not have the heketi-client rpm installed on any OCP nodes:

    # yum update heketi-client -y
  3. Execute the following command to get the current HEKETI_ADMIN_KEY.

    The OCS admin can choose to set any phrase for user key as long as it is not used by their infrastructure. It is not used by any of the OCS default installed resources.

    # oc get secret heketi-storage-admin-secret -o jsonpath='{.data.key}'|base64 -d;echo
  4. If the HEKETI_USER_KEY was set previously, you can obtain it by using the following command:

    # oc describe pod <heketi-pod>
  5. Execute the following command to delete the heketi template.

    # oc delete templates heketi
  6. Execute the following command to install the heketi template.

    # oc create -f /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/heketi-template.yml
    template "heketi" created
  7. Execute the following step to edit the template:

    # oc get templates
    NAME			  DESCRIPTION		     PARAMETERS		OBJECTS
    glusterblock-provisioner  glusterblock provisioner   3 (2 blank)	4
    			  template
    glusterfs		  GlusterFS DaemonSet 	     5 (1 blank)	1
    			  template
    heketi			  Heketi service deployment  7 (3 blank)	3
    template
    1. If the existing template has IMAGE_NAME and IMAGE_VERSION as two parameters, then edit the template to change the HEKETI_USER_KEY, HEKETI_ADMIN_KEY, HEKETI_ROUTE, IMAGE_NAME, IMAGE_VERSION, CLUSTER_NAME and HEKETI_LVM_WRAPPER as shown in the example below.

      # oc edit template heketi
      parameters:
        - description: Set secret for those creating volumes as type user
        displayName: Heketi User Secret
        name: HEKETI_USER_KEY
        value: <heketiuserkey>
        - description: Set secret for administration of the Heketi service as user admin
        displayName: Heketi Administrator Secret
        name: HEKETI_ADMIN_KEY
        value: <adminkey>
        - description: Set the executor type, kubernetes or ssh
        displayName: heketi executor type
        name: HEKETI_EXECUTOR
        value: kubernetes
        - description: Set the hostname for the route URL
        displayName: heketi route name
        name: HEKETI_ROUTE
        value: heketi-storage
        - displayName: heketi container image name
        name: IMAGE_NAME
        required: true
        value: registry.redhat.io/rhgs3/rhgs-volmanager-rhel7
        - displayName: heketi container image version
        name: IMAGE_VERSION
        required: true
        value: v3.11.6
        - description: A unique name to identify this heketi service, useful for running
        multiple heketi instances
        displayName: GlusterFS cluster name
        name: CLUSTER_NAME
        value: storage
        - description: Heketi can use a wrapper to execute LVM commands, i.e. run commands in the host namespace instead of in the Gluster container
        name: HEKETI_LVM_WRAPPER
        displayName: Wrapper for executing LVM commands
        value: /usr/sbin/exec-on-host
    2. If the template has only IMAGE_NAME, then edit the template to change the HEKETI_USER_KEY, HEKETI_ADMIN_KEY, HEKETI_ROUTE, IMAGE_NAME, CLUSTER_NAME and HEKETI_LVM_WRAPPER as shown in the example below.

      # oc edit template heketi
      parameters:
      - description: Set secret for those creating volumes as type user
        displayName: Heketi User Secret
        name: HEKETI_USER_KEY
        value: <heketiuserkey>
      - description: Set secret for administration of the Heketi service as user admin
        displayName: Heketi Administrator Secret
        name: HEKETI_ADMIN_KEY
        value: <adminkey>
      - description: Set the executor type, kubernetes or ssh
        displayName: heketi executor type
        name: HEKETI_EXECUTOR
        value: kubernetes
      - description: Set the hostname for the route URL
        displayName: heketi route name
        name: HEKETI_ROUTE
        value: heketi-storage
      - displayName: heketi container image name
        name: IMAGE_NAME
        required: true
        value: registry.redhat.io/rhgs3/rhgs-volmanager-rhel7:v3.11.6
      - description: A unique name to identify this heketi service, useful for running
         multiple heketi instances
        displayName: GlusterFS cluster name
        name: CLUSTER_NAME
        value: storage
      - description: Heketi can use a wrapper to execute LVM commands, i.e. run commands in the host namespace instead of in the Gluster container
        name: HEKETI_LVM_WRAPPER
        displayName: Wrapper for executing LVM commands
        value: /usr/sbin/exec-on-host
      Note

      If a cluster has more than 1000 volumes refer to How to change the default PVS limit in Openshift Container Storage and add the required parameters before proceeding with the upgrade.

  8. Execute the following command to delete the deployment configuration, service, and route for heketi:

    Note

    The names of these parameters can be referenced from output of the following command:

    # oc get all | grep heketi
    # oc delete deploymentconfig,service,route heketi-storage
  9. Execute the following command to deploy the Heketi service, route, and deployment configuration which will be used to create persistent volumes for OpenShift:

    # oc process heketi | oc create -f -
    
    service "heketi" created
    route "heketi" created
    deploymentconfig "heketi" created
    Note

    It is recommended that the heketidbstorage volume be tuned for db workloads. Newly installed Openshift Container Storage deployments tune the heketidbstorage volume automatically. For older deployments, follow the KCS article Planning to run containerized DB or nosql workloads on Openshift Container Storage? and perform the volume set operation for the volume heketidbstorage.

  10. Execute the following command to verify that the containers are running:

    # oc get pods

    For example:

    # oc get pods
      NAME                             READY     STATUS    RESTARTS   AGE
      glusterfs-0h68l                  1/1       Running   0          3d
      glusterfs-0vcf3                  1/1       Running   0          3d
      glusterfs-gr9gh                  1/1       Running   0          3d
      heketi-1-zpw4d                   1/1       Running   0          3h
      storage-project-router-2-db2wl   1/1       Running   0          4d

6.1.4.2. Upgrading the Red Hat Gluster Storage Pods

The following commands must be executed on the client machine.

Following are the steps for updating a DaemonSet for glusterfs:

  1. Execute the following steps to stop the Heketi pod to prevent it from accepting any new request for volume creation or volume deletion:

    1. Execute the following command to access your project:

      # oc project <project_name>

      For example:

      # oc project storage-project
    2. Execute the following command to get the DeploymentConfig:

      # oc get dc
    3. Execute the following command to set heketi server to accept requests only from the local-client:

      # heketi-cli server mode set local-client
    4. Wait for the ongoing operations to complete and execute the following command to monitor if there are any ongoing operations:

      # heketi-cli server operations info
    5. Execute the following command to reduce the replica count from 1 to 0. This brings down the Heketi pod:

      # oc scale dc <heketi_dc> --replicas=0
    6. Execute the following command to verify that the heketi pod is no longer present:

      # oc get pods
  2. Execute the following command to find the DaemonSet name for gluster

    # oc get ds
  3. Execute the following command to delete the DaemonSet:

    # oc delete ds <ds-name> --cascade=false

    Using --cascade=false option while deleting the old DaemonSet does not delete the gluster pods but deletes only the DaemonSet. After deleting the old DaemonSet, you must load the new one. When you manually delete the old pods, the new pods which are created will have the configurations of the new DaemonSet.

    For example,

    # oc delete ds glusterfs-storage --cascade=false
    daemonset "glusterfs-storage" deleted
  4. Execute the following commands to verify all the old pods are up:

    # oc get pods

    For example,

    # oc get pods
    NAME                             READY     STATUS    RESTARTS   AGE
    glusterfs-0h68l                  1/1       Running   0          3d
    glusterfs-0vcf3                  1/1       Running   0          3d
    glusterfs-gr9gh                  1/1       Running   0          3d
    storage-project-router-2-db2wl   1/1       Running   0          4d
  5. Execute the following command to delete the old glusterfs template.

    # oc delete templates glusterfs
  6. Execute the following command to register new glusterfs template.

    # oc create -f /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/glusterfs-template.yml
    template "glusterfs" created
  7. Execute the following command to edit the old glusterfs template.

    # oc get templates
    NAME			  DESCRIPTION		     PARAMETERS		OBJECTS
    glusterblock-provisioner  glusterblock provisioner   3 (2 blank)	4
            template
    glusterfs		  GlusterFS DaemonSet 	     5 (1 blank)	1
            template
    heketi			  Heketi service deployment  7 (3 blank)	3
    template
    1. If the template has IMAGE_NAME and IMAGE_VERSION as two separate parameters, then update the glusterfs template as following. For example:

      # oc edit template glusterfs
      - displayName: GlusterFS container image name
      	  name: IMAGE_NAME
      	  required: true
      	  value: registry.redhat.io/rhgs3/rhgs-server-rhel7
      - displayName: GlusterFS container image version
      	  name: IMAGE_VERSION
      	  required: true
      	  value: v3.11.6
      - description: A unique name to identify which heketi service manages this cluster, useful for running
           multiple heketi instances
        displayName: GlusterFS cluster name
        name: CLUSTER_NAME
        value: storage
      Note

      If a cluster has more than 1000 volumes refer to How to change the default PVS limit in Openshift Container Storage and add the required parameters before proceeding with the upgrade.

    2. If the template has only IMAGE_NAME as a parameter, then update the glusterfs template as following. For example:

      # oc edit template glusterfs
      - displayName: GlusterFS container image name
        name: IMAGE_NAME
        required: true
        value: registry.redhat.io/rhgs3/rhgs-server-rhel7:v3.11.6
      - description: A unique name to identify which heketi service manages this cluster, useful for running
           multiple heketi instances
        displayName: GlusterFS cluster name
        name: CLUSTER_NAME
      value: storage
      Note

      Ensure that the CLUSTER_NAME variable is set to the correct value

  8. Label all the OpenShift Container Platform nodes that has the Red Hat Gluster Storage pods:

    1. Check if the nodes are labelled with the appropriate label by using the following command:

      # oc get nodes -l glusterfs=storage-host
  9. Execute the following commands to create the gluster DaemonSet:

    # oc process glusterfs | oc create -f -

    For example,

    # oc process glusterfs | oc create -f -
    Deamonset “glusterfs” created
    • Execute the following command to identify the old gluster pods that needs to be deleted:

      # oc get pods

      For example,

      # oc get pods
      NAME                             READY     STATUS    RESTARTS   AGE
      glusterfs-0h68l                  1/1       Running   0          3d
      glusterfs-0vcf3                  1/1       Running   0          3d
      glusterfs-gr9gh                  1/1       Running   0          3d
      storage-project-router-2-db2wl   1/1       Running   0          4d
  10. Execute the following command and ensure that the bricks are not more than 90% full:

    # df -kh | grep -v ^Filesystem | awk '{if(int($5)>90) print $0}'
    Note

    If the bricks are close to 100% utilization, then the Logical Volume Manager(LVM) activation for these bricks may take a long time or can get stuck once the pod or node is rebooted. It is advised to bring down the utilization of that brick or expand the physical volume(PV) that is using the logical volume(LV).

  11. Execute the following command to delete the old gluster pods. Gluster pods should follow rolling upgrade. Hence, you must ensure that the new pod is running before deleting the next old gluster pod. We support OnDelete Strategy DaemonSet update strategy. With OnDelete Strategy update strategy, after you update a DaemonSet template, new DaemonSet pods will only be created when you manually delete old DaemonSet pods.

    1. To delete the old gluster pods, execute the following command:

      # oc delete pod <gluster_pod>

      For example,

      # oc delete pod glusterfs-0vcf3
      pod  “glusterfs-0vcf3” deleted
      Note

      Before deleting the next pod, self heal check has to be made:

      1. Run the following command to access shell on gluster pod:

        # oc rsh <gluster_pod_name>
      2. Run the following command to check the self-heal status of all the volumes:

        for each_volume in `gluster volume list`;
            do gluster volume heal $each_volume info ;
        done | grep "Number of entries: [^0]$"
    2. The delete pod command will terminate the old pod and create a new pod. Run # oc get pods -w and check the Age of the pod and READY status should be 1/1. The following is the example output showing the status progression from termination to creation of the pod.

      # oc get pods -w
      NAME                             READY     STATUS        RESTARTS   AGE
      glusterfs-0vcf3                  1/1       Terminating   0          3d
      …
  12. Execute the following command to verify that the pods are running:

    # oc get pods

    For example,

    # oc get pods
    NAME                             READY     STATUS    RESTARTS   AGE
    glusterfs-j241c                  1/1       Running   0          4m
    glusterfs-pqfs6                  1/1       Running   0          7m
    glusterfs-wrn6n                  1/1       Running   0          12m
    storage-project-router-2-db2wl   1/1       Running   0          4d
  13. Execute the following command to verify if you have upgraded the pod to the latest version:

    # oc rsh <gluster_pod_name> glusterd --version

    For example:

    # oc rsh glusterfs-4cpcc glusterd --version
    glusterfs 6.0
  14. Check the Red Hat Gluster Storage op-version by executing the following command on one of the gluster pods.

    # gluster vol get all cluster.op-version
  15. After you upgrade the Gluster pods, ensure that you set Heketi back to operational mode:

    • Scale up the DC (Deployment Configuration).

      # oc scale dc <heketi_dc> --replicas=1
  16. Set the cluster.op-version to 70000 on any one of the pods:

    Note

    Ensure all the gluster pods are updated before changing the cluster.op-version.

    # gluster --timeout=3600 volume set all cluster.op-version 70000
  17. Execute the following steps to enable server.tcp-user-timeout on all volumes.

    Note

    The "server.tcp-user-timeout" option specifies the maximum amount of the time (in seconds) the transmitted data from the application can remain unacknowledged from the brick.

    It is used to detect force disconnections and dead connections (if a node dies unexpectedly, a firewall is activated, etc.,) early and make it possible for applications to reduce the overall failover time.

    1. List the glusterfs pod using the following command:

      # oc get pods

      For example:

      # oc get pods
          NAME                             READY     STATUS    RESTARTS   AGE
          glusterfs-0h68l                  1/1       Running   0          3d
          glusterfs-0vcf3                  1/1       Running   0          3d
          glusterfs-gr9gh                  1/1       Running   0          3d
      storage-project-router-2-db2wl   1/1       Running   0          4d
    2. Remote shell into one of the glusterfs pods. For example:

      # oc rsh glusterfs-0vcf3
    3. Execute the following command:

      # for eachVolume in `gluster volume list`; do echo $eachVolume; gluster volume set $eachVolume server.tcp-user-timeout 42 ; done

      For example:

      # for eachVolume in `gluster volume list`; do echo $eachVolume; gluster volume set $eachVolume server.tcp-user-timeout 42 ; done
          volume1
          volume set: success
          volume2
      volume set: success
  18. If a gluster-block-provisoner-pod already exists then delete it by executing the following commands:

    # oc delete dc glusterblock-provisioner-dc

    For example:

    # oc delete dc glusterblock-storage-provisioner-dc
  19. Execute the following command to delete the old glusterblock provisioner template.

     # oc delete templates glusterblock-provisioner
  20. Execute the following command to register new glusterblock provisioner template, see Templates on GitHub. Copy and paste to new-block-prov.yaml. For example,

    # oc create -f new-block-prov.yaml
    template.template.openshift.io/glusterblock-provisioner created
  21. Depending on the OCP version, edit the glusterblock-provisioner template to change the IMAGE_NAME, IMAGE_VERSION and NAMESPACE.

    # oc get templates
    NAME			  DESCRIPTION		     PARAMETERS		OBJECTS
    glusterblock-provisioner  glusterblock provisioner   3 (2 blank)	4
            template
    glusterfs		  GlusterFS DaemonSet 	     5 (1 blank)	1
            template
    heketi			  Heketi service deployment  7 (3 blank)	3
    template
    1. If the template has IMAGE_NAME and IMAGE_VERSION as two separate parameters, then update the glusterblock-provisioner template as following. For example:

      # oc edit template glusterblock-provisioner
      - displayName: glusterblock provisioner container image name
      name: IMAGE_NAME
      required: true
      value: registry.redhat.io/rhgs3/rhgs-gluster-block-prov-rhel7
      - displayName: glusterblock provisioner container image version
      name: IMAGE_VERSION
      required: true
      value: v3.11.6
      - description: The namespace in which these resources are being created
      displayName: glusterblock provisioner namespace
      name: NAMESPACE
      required: true
      value: glusterfs
      - description: A unique name to identify which heketi service manages this cluster,
        useful for running multiple heketi instances
      displayName: GlusterFS cluster name
      name: CLUSTER_NAME
      value: storage
    2. If the template has only IMAGE_NAME as a parameter, then update the glusterblock-provisioner template as following. For example:

      # oc edit template glusterblock-provisioner
      - displayName: glusterblock provisioner container image name
      name: IMAGE_NAME
      required: true
      value: registry.redhat.io/rhgs3/rhgs-gluster-block-prov-rhel7:v3.11.6
      - description: The namespace in which these resources are being created
      displayName: glusterblock provisioner namespace
      name: NAMESPACE
      required: true
      value: glusterfs
      - description: A unique name to identify which heketi service manages this cluster,
                  useful for running multiple heketi instances
      displayName: GlusterFS cluster name
      name: CLUSTER_NAME
      value: storage
  22. Delete the following resources from the old pod

    # oc delete clusterroles.authorization.openshift.io glusterblock-provisioner-runner
    # oc delete serviceaccounts glusterblock-storage-provisioner
    # oc delete clusterrolebindings.authorization.openshift.io glusterblock-storage-provisioner
  23. Before running oc process determine the correct provisioner name. If there are more than one gluster block provisioner running in your cluster the names must differ from all other provisioners.
    For example,

    • If there are 2 or more provisioner the name should be gluster.org/glusterblock-<namespace> where, namespace is replaced by the namespace that the provisioner is deployed in.
    • If there is only one provisioner, installed prior to 3.11.5, gluster.org/glusterblock is sufficent. If the name currently in use already has a unique namespace suffix, reuse the existing name.
  24. After editing the template, execute the following command to create the deployment configuration:

    # oc process -p PROVISIONER_NAME=<provisioner-name> glusterblock-provisioner -o yaml | oc create -f -

    For example:

     # oc process -p PROVISIONER_NAME=gluster.org/glusterblock-app-storage glusterblock-provisioner -o yaml | oc create -f -
     clusterrole.authorization.openshift.io/glusterblock-provisioner-runner created
     serviceaccount/glusterblock-storage-provisioner created
     clusterrolebinding.authorization.openshift.io/glusterblock-storage-provisioner created
     deploymentconfig.apps.openshift.io/glusterblock-storage-provisioner-dc created
  25. Brick multiplexing is a feature that allows adding multiple bricks into one process. This reduces resource consumption and allows us to run more bricks than before with the same memory consumption. It is enabled by default from Container-Native Storage 3.6 onward. During an upgrade from Container-Native Storage 3.10 to Red Hat Openshift Container Storage 3.11, to turn brick multiplexing on, execute the following commands:

    1. To exec into the Gluster pod, execute the following command and rsh into any of the gluster pods:

      # oc rsh <gluster_pod_name>
    2. Verify the brick multiplex status:

      # gluster v get all all
    3. If it is disabled, then execute the following command to enable brick multiplexing:

      Note

      Ensure that all volumes are in a stop state or no bricks are running while brick multiplexing is enabled.

      # gluster volume set all cluster.brick-multiplex on

      For example:

      # oc rsh glusterfs-770ql
      sh-4.2# gluster volume set all cluster.brick-multiplex on
      Brick-multiplexing is supported only for container workloads (Independent or Converged mode). Also it is advised to make sure that either all volumes are in stopped state or no bricks are running before this option is modified.Do you still want to continue? (y/n) y
      volume set: success
    4. List all the volumes in the trusted storage pool. This step is only required if the volume set operation is performed:

      For example:

      # gluster volume list
      
      heketidbstorage
      vol_194049d2565d2a4ad78ef0483e04711e
      ...
      ...

      Restart all the volumes. This step is only required if the volume set operation is performed along with the previous step:

      # gluster vol stop <VOLNAME>
      # gluster vol start <VOLNAME>
  26. Support for S3 compatible Object Store in Red Hat Openshift Container Storage is under technology preview. To enable S3 compatible object store, see https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html/operations_guide/s3_object_store.

    Note
  27. All storage classes that use gluster block volume provisioning must match exactly to one of the provisioner names in the cluster. To check the list of storage classes that refer to a block provisioner, in a given namespace, run the following command:

    # oc get sc -o custom-columns=NAME:.metadata.name,PROV:.provisioner,RSNS:.parameters.restsecretnamespace | grep 'gluster.org/glusterblock' | grep <namespace>

    Example:

    # oc get sc -o custom-columns=NAME:.metadata.name,PROV:.provisioner,RSNS:.parameters.restsecretnamespace | grep 'gluster.org/glusterblock' | grep app-storage
    glusterfs-storage-block   gluster.org/glusterblock-app-storage   app-storage

    Check each storage class provisioner name, if it does not match the block provisioner name configured for that namespace it must be updated. If the block provisioner name already matches the configured provisioner name, nothing else needs to be done. Use the list generated above and include all storage class names where the provionser name must be updated.
    For every storage class in this list do the following:

    # oc get sc  -o yaml <storageclass>  > storageclass-to-edit.yaml
    # oc delete sc  <storageclass>
    # sed 's,gluster.org/glusterblock$,gluster.org/glusterblock-<namespace>,' storageclass-to-edit.yaml | oc create -f -

    Example:

    # oc get sc  -o yaml gluster-storage-block  > storageclass-to-edit.yaml
    # oc delete sc  gluster-storage-block
    # sed 's,gluster.org/glusterblock$,gluster.org/glusterblock-app-storage,' storageclass-to-edit.yaml | oc create -f -

6.2. Upgrading the pods in the glusterfs registry group

The following sections provide steps to upgrade your glusterfs registry pods.

6.2.1. Prerequisites

Ensure the following prerequisites are met:

Note

For deployments using cns-deploy tool, the templates are available in the following location:

  • gluster template - /usr/share/heketi/templates/glusterfs-template.yaml
  • heketi template - /usr/share/heketi/templates/heketi-template.yaml
  • glusterblock-provisioner template - /usr/share/heketi/templates/glusterblock-provisioner.yaml

For deployments using ansible playbook the templates are available in the following location:

  • gluster template - /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/glusterfs-template.yml
  • heketi template - /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/heketi-template.yml
  • glusterblock-provisioner template - /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/glusterblock-provisioner.yml

6.2.2. Restoring original label values for /dev/log

Note

Follow this procedure only if you are upgrading your environment from Red Hat Container Native Storage 3.9 to Red Hat Openshift Container Storage 3.11.5.

Skip this procedure if you are upgrading your environment from Red Hat Openshift Container Storage 3.10 and above to Red Hat Openshift Container Storage 3.11.5.

To restore the original selinux label, execute the following commands:

  1. Create a directory and soft links on all nodes that run gluster pods:

    # mkdir /srv/<directory_name>
    # cd /srv/<directory_name>/   # same dir as above
    # ln -sf /dev/null systemd-tmpfiles-setup-dev.service
    # ln -sf /dev/null systemd-journald.service
    # ln -sf /dev/null systemd-journald.socket
  2. Edit the daemonset that creates the glusterfs pods on the node which has oc client:

    # oc edit daemonset <daemonset_name>

    Under volumeMounts section add a mapping for the volume:

    - mountPath: /usr/lib/systemd/system/systemd-journald.service
      name: systemd-journald-service
    - mountPath: /usr/lib/systemd/system/systemd-journald.socket
      name: systemd-journald-socket
    - mountPath: /usr/lib/systemd/system/systemd-tmpfiles-setup-dev.service
    name: systemd-tmpfiles-setup-dev-service

    Under volumes section add a new host path for each service listed:

    Note

    The path mentioned in here should be the same as mentioned in Step 1.

    - hostPath:
       path: /srv/<directory_name>/systemd-journald.socket
       type: ""
      name: systemd-journald-socket
    - hostPath:
       path: /srv/<directory_name>/systemd-journald.service
       type: ""
      name: systemd-journald-service
    - hostPath:
       path: /srv/<directory_name>/systemd-tmpfiles-setup-dev.service
       type: ""
    name: systemd-tmpfiles-setup-dev-service
  3. Run the following command on all nodes that run gluster pods. This will reset the label:

    # restorecon /dev/log
  4. Execute the following command to check the status of self heal for all volumes:

    # oc rsh <gluster_pod_name>
    # for each_volume in `gluster volume list`; do gluster volume heal $each_volume info ; done  | grep  "Number of entries: [^0]$"

    Wait for self-heal to complete.

  5. Execute the following command and ensure that the bricks are not more than 90% full:

    # df -kh | grep -v ^Filesystem | awk '{if(int($5)>90) print $0}'
    Note

    If the bricks are close to 100% utilization, then the Logical Volume Manager(LVM) activation for these bricks may take a long time or can get stuck once the pod or node is rebooted. It is advised to bring down the utilization of that brick or expand the physical volume(PV) that is using the logical volume(LV).

  6. Execute the following command on any one of the gluster pods to set the maximum number of bricks (250) that can run on a single instance of glusterfsd process:

    # gluster volume set all cluster.max-bricks-per-process 250
    1. Execute the following command on any one of the gluster pods to ensure that the option is set correctly:

      # gluster volume get all cluster.max-bricks-per-process

      For example:

      # gluster volume get all cluster.max-bricks-per-process
      cluster.max-bricks-per-process 250
  7. Execute the following command on the node which has oc client to delete the gluster pod:

    # oc delete pod <gluster_pod_name>
  8. To verify if the pod is ready, execute the following command:

    # oc get pods -l glusterfs=registry-pod
  9. Login to the node hosting the pod and check the selinux label of /dev/log

    # ls -lZ /dev/log

    The output should show devlog_t label

    For example:

    #  ls -lZ /dev/log
    srw-rw-rw-. root root system_u:object_r:devlog_t:s0    /dev/log

    Exit the node.

  10. In the gluster pod, check if the label value is devlog_t:

    # oc rsh <gluster_pod_name>
    # ls -lZ /dev/log

    For example:

    #  ls -lZ /dev/log
    srw-rw-rw-. root root system_u:object_r:devlog_t:s0    /dev/log
  11. Perform steps 4 to 9 for other pods.

6.2.3. Upgrading if existing version deployed by using cns-deploy

6.2.3.1. Upgrading cns-deploy and Heketi Server

The following commands must be executed on the client machine.

  1. Execute the following command to update the heketi client and cns-deploy packages:

    # yum update cns-deploy -y
    # yum update heketi-client -y
  2. Backup the Heketi registry database file

    # heketi-cli db dump > heketi-db-dump-$(date -I).json
  3. Execute the following command to delete the heketi template.

    # oc delete templates heketi
  4. Execute the following command to get the current HEKETI_ADMIN_KEY.

    The OCS admin can choose to set any phrase for user key as long as it is not used by their infrastructure. It is not used by any of the OCS default installed resources.

    # oc get secret <heketi-admin-secret-name> -o jsonpath='{.data.key}'|base64 -d;echo
  5. Execute the following command to install the heketi template.

    # oc create -f /usr/share/heketi/templates/heketi-template.yaml
    template "heketi" created
  6. Execute the following command to grant the heketi Service Account the necessary privileges.

    # oc policy add-role-to-user edit system:serviceaccount:<project_name>:heketi-service-account
    # oc adm policy add-scc-to-user privileged -z heketi-service-account

    For example,

    # oc policy add-role-to-user edit system:serviceaccount:storage-project:heketi-service-account
    # oc adm policy add-scc-to-user privileged -z heketi-service-account
    Note

    The service account used in heketi pod needs to be privileged because Heketi/rhgs-volmanager pod mounts the heketidb storage Gluster volume as a "glusterfs" volume type and not as a PersistentVolume (PV).
    As per the security-context-constraints regulations in OpenShift, ability to mount volumes which are not of the type configMap, downwardAPI, emptyDir, hostPath, nfs, persistentVolumeClaim, secret is granted only to accounts with privileged Security Context Constraint (SCC).

  7. Execute the following command to generate a new heketi configuration file.

    # sed -e "s/\${HEKETI_EXECUTOR}/kubernetes/" -e "s#\${HEKETI_FSTAB}#/var/lib/heketi/fstab#" -e "s/\${SSH_PORT}/22/" -e "s/\${SSH_USER}/root/" -e "s/\${SSH_SUDO}/false/" -e "s/\${BLOCK_HOST_CREATE}/true/" -e "s/\${BLOCK_HOST_SIZE}/500/" "/usr/share/heketi/templates/heketi.json.template" > heketi.json
    • The BLOCK_HOST_SIZE parameter controls the size (in GB) of the automatically created Red Hat Gluster Storage volumes hosting the gluster-block volumes (For more information, see https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html-single/operations_guide/index#Block_Storage). This default configuration will dynamically create block-hosting volumes of 500GB in size as more space is required.
    • Alternatively, copy the file /usr/share/heketi/templates/heketi.json.template to heketi.json in the current directory and edit the new file directly, replacing each "${VARIABLE}" string with the required parameter.

      Note

      JSON formatting is strictly required (e.g. no trailing spaces, booleans in all lowercase).

  8. Execute the following command to create a secret to hold the configuration file.

    # oc create secret generic <heketi-registry-config-secret> --from-file=heketi.json
    Note

    If the heketi-registry-config-secret file already exists, then delete the file and run the following command.

  9. Execute the following command to delete the deployment configuration, service, and route for heketi:

    # oc delete deploymentconfig,service,route heketi-registry
  10. Execute the following command to edit the heketi template. Edit the HEKETI_USER_KEY and HEKETI_ADMIN_KEY parameters.

    # oc edit template heketi
    parameters:
    - description: Set secret for those creating volumes as type user
      displayName: Heketi User Secret
      name: HEKETI_USER_KEY
      value: <heketiuserkey>
    - description: Set secret for administration of the Heketi service as user admin
      displayName: Heketi Administrator Secret
      name: HEKETI_ADMIN_KEY
      value: <adminkey>
    - description: Set the executor type, kubernetes or ssh
      displayName: heketi executor type
      name: HEKETI_EXECUTOR
      value: kubernetes
    - description: Set the hostname for the route URL
      displayName: heketi route name
      name: HEKETI_ROUTE
      value: heketi-storage
    - displayName: heketi container image name
      name: IMAGE_NAME
      required: true
      value: registry.redhat.io/rhgs3/rhgs-volmanager-rhel7
    - displayName: heketi container image version
      name: IMAGE_VERSION
      required: true
      value: v3.11.6
    - description: A unique name to identify this heketi service, useful for running
        multiple heketi instances
      displayName: GlusterFS cluster name
      name: CLUSTER_NAME
      value: storage
    Note

    If a cluster has more than 1000 volumes refer to How to change the default PVS limit in Openshift Container Storage and add the required parameters before proceeding with the upgrade.

  11. Execute the following command to deploy the Heketi service, route, and deployment configuration which will be used to create persistent volumes for OpenShift:

    # oc process heketi | oc create -f -
    
    service "heketi-registry" created
    route "heketi-registry" created
    deploymentconfig-registry "heketi" created
    Note

    It is recommended that the heketidbstorage volume be tuned for db workloads. Newly installed Openshift Container Storage deployments tune the heketidbstorage volume automatically. For older deployments, follow the KCS article Planning to run containerized DB or nosql workloads on Openshift Container Storage? and perform the volume set operation for the volume heketidbstorage.

  12. Execute the following command to verify that the containers are running:

    # oc get pods

    For example:

    # oc get pods
    NAME                                           READY     STATUS    RESTARTS   AGE
    glusterblock-registry-provisioner-dc-1-lm7ht   1/1       Running   81         14d
    glusterfs-registry-l25b9                       1/1       Running   10         14d
    glusterfs-registry-vl6qs                       1/1       Running   10         14d
    glusterfs-registry-zhxwg                       1/1       Running   10         14d
    heketi-registry-1-54bwf                        1/1       Running   10         14d

6.2.3.2. Upgrading the Red Hat Gluster Storage Registry Pods

The following commands must be executed on the client machine. .

Following are the steps for updating a DaemonSet for glusterfs:

  1. Execute the following steps to stop the Heketi pod to prevent it from accepting any new request for volume creation or volume deletion:

    1. Execute the following command to access your project:

      # oc project <project_name>

      For example:

      # oc project storage-project
    2. Execute the following command to get the DeploymentConfig:

      # oc get ds
    3. Execute the following command to set heketi server to accept requests only from the local-client:

      # heketi-cli server mode set local-client
    4. Wait for the ongoing operations to complete and execute the following command to monitor if there are any ongoing operations:

      # heketi-cli server operations info
    5. Execute the following command to reduce the replica count from 1 to 0. This brings down the Heketi pod:

      # oc scale dc <heketi_dc> --replicas=0
    6. Execute the following command to verify that the heketi pod is no longer present:

      # oc get pods
  2. Execute the following command to find the DaemonSet name for gluster

    # oc get ds
  3. Execute the following command to delete the DaemonSet:

    # oc delete ds <ds-name> --cascade=false

    Using --cascade=false option while deleting the old DaemonSet does not delete the glusterfs_registry pods but deletes only the DaemonSet. After deleting the old DaemonSet, you must load the new one. When you manually delete the old pods, the new pods which are created will have the configurations of the new DaemonSet.

    For example,

    # oc delete ds glusterfs-registry --cascade=false
    daemonset "glusterfs-registry" deleted
  4. Execute the following commands to verify all the old pods are up:

    # oc get pods

    For example,

    # oc get pods
    NAME                                           READY     STATUS    RESTARTS   AGE
    glusterblock-registry-provisioner-dc-1-nvnhc   1/1       Running   1          7d
    glusterfs-registry-4cpcc                       1/1       Running   0          7d
    glusterfs-registry-9xj78                       1/1       Running   0          7d
    glusterfs-registry-b9p5j                       1/1       Running   0          7d
  5. Execute the following command to delete the old glusterfs template.

    # oc delete templates glusterfs

    For example,

    # oc delete templates glusterfs
    template “glusterfs” deleted
  6. Label all the OpenShift Container Platform nodes that has the Red Hat Gluster Storage pods:

    1. Check if the nodes are labelled with the appropriate label by using the following command:

      # oc get nodes -l glusterfs=registry-host
  7. Execute the following command to register new glusterfs template.

    # oc create -f /usr/share/heketi/templates/glusterfs-template.yaml

    For example,

    # oc create -f /usr/share/heketi/templates/glusterfs-template.yaml
    template “glusterfs” created
  8. Execute the following commands to create the gluster DaemonSet:

    # oc process glusterfs | oc create -f -

    For example,

    # oc process glusterfs | oc create -f -
    Deamonset “glusterfs” created
    Note

    If a cluster has more than 1000 volumes refer to How to change the default PVS limit in Openshift Container Storage and add the required parameters before proceeding with the upgrade.

  9. Execute the following command to identify the old glusterfs_registry pods that needs to be deleted:

    # oc get pods

    For example,

    # oc get pods
    NAME                                           READY     STATUS    RESTARTS   AGE
    glusterblock-registry-provisioner-dc-1-nvnhc   1/1       Running   1          7d
    glusterfs-registry-4cpcc                       1/1       Running   0          7d
    glusterfs-registry-9xj78                       1/1       Running   0          7d
    glusterfs-registry-b9p5j                       1/1       Running   0          7d
  10. Execute the following command and ensure that the bricks are not more than 90% full:

    # df -kh | grep -v ^Filesystem | awk '{if(int($5)>90) print $0}'
    Note

    If the bricks are close to 100% utilization, then the Logical Volume Manager(LVM) activation for these bricks may take a long time or can get stuck once the pod or node is rebooted. It is advised to bring down the utilization of that brick or expand the physical volume(PV) that is using the logical volume(LV).

  11. Execute the following command to delete the old glusterfs-registry pods. glusterfs-registry pods should follow rolling upgrade. Hence, you must ensure that the new pod is running before deleting the next old glusterfs-registry pods. We support OnDelete Strategy DaemonSet update strategy. With OnDelete Strategy update strategy, after you update a DaemonSet template, new DaemonSet pods will only be created when you manually delete old DaemonSet pods.

    1. To delete the old glusterfs-registry pods, execute the following command:

      # oc delete pod <gluster_pod>

      For example,

      # oc delete pod glusterfs-0vcf3
      pod  “glusterfs-0vcf3” deleted
      Note

      Before deleting the next pod, self heal check has to be made:

      1. Run the following command to access shell on glusterfs-registry pods:

        # oc rsh <gluster_pod_name>
      2. Run the following command to check the self-heal status of all the volumes: :

        # for each_volume in `gluster volume list`; do gluster volume heal $each_volume info ; done | grep "Number of entries: [^0]$"
    2. The delete pod command will terminate the old pod and create a new pod. Run # oc get pods -w and check the Age of the pod and READY status should be 1/1. The following is the example output showing the status progression from termination to creation of the pod.

      # oc get pods -w
      NAME                             READY     STATUS        RESTARTS   AGE
      glusterfs-0vcf3                  1/1       Terminating   0          3d
      …
  12. Execute the following command to verify that the pods are running:

    # oc get pods

    For example,

    # oc get pods
    NAME                                           READY     STATUS    RESTARTS   AGE
    glusterblock-registry-provisioner-dc-1-nvnhc   1/1       Running   1          7d
    glusterfs-registry-4cpcc                       1/1       Running   0          7d
    glusterfs-registry-9xj78                       1/1       Running   0          7d
    glusterfs-registry-b9p5j                       1/1       Running   0          7d
    • Execute the following commands to verify if you have upgraded the pod to the latest version:

      # oc rsh <gluster_registry_pod_name> glusterd --version

      For example:

       # oc rsh glusterfs-registry-4cpcc glusterd --version
      glusterfs 6.0
      # rpm -qa|grep gluster
  13. Check the Red Hat Gluster Storage op-version by executing the following command on one of the glusterfs-registry pods.

    # gluster vol get all cluster.op-version
  14. After you upgrade the Gluster pods, ensure that you set Heketi back to operational mode:

    • Scale up the DC (Deployment Configuration).

      # oc scale dc <heketi_dc> --replicas=1
  15. Set the cluster.op-version to 70000 on any one of the pods:

    Note

    Ensure all the glusterfs-registry pods are updated before changing the cluster.op-version.

    # gluster volume set all cluster.op-version 70000
  16. Execute the following steps to enable server.tcp-user-timeout on all volumes.

    Note

    The "server.tcp-user-timeout" option specifies the maximum amount of the time (in seconds) the transmitted data from the application can remain unacknowledged from the brick.

    It is used to detect force disconnections and dead connections (if a node dies unexpectedly, a firewall is activated, etc.,) early and make it possible for applications to reduce the overall failover time.

    1. List the glusterfs pod using the following command:

      # oc get pods

      For example:

      # oc get pods
      NAME                                           READY     STATUS    RESTARTS   AGE
      glusterblock-registry-provisioner-dc-1-nvnhc   1/1       Running   1          7d
      glusterfs-registry-4cpcc                       1/1       Running   0          7d
      glusterfs-registry-9xj78                       1/1       Running   0          7d
      glusterfs-registry-b9p5j                       1/1       Running   0          7d
    2. Remote shell into one of the glusterfs-registry pods. For example:

      # oc rsh glusterfs-registry-g6vd9
    3. Execute the following command:

      # for eachVolume in `gluster volume list`; do echo $eachVolume; gluster volume set $eachVolume server.tcp-user-timeout 42 ; done

      For example:

      # for eachVolume in `gluster volume list`; do echo $eachVolume; gluster volume set $eachVolume server.tcp-user-timeout 42 ; done
      volume1
      volume set: success
      volume2
      volume set: success
  17. If a gluster-block-registry-provisoner-pod already exists then delete it by executing the following commands:

    # oc delete dc <gluster-block-registry-dc>

    For example:

    # oc delete dc glusterblock-registry-provisioner-dc
  18. Delete the following resources from the old pod

    # oc delete clusterroles.authorization.openshift.io glusterblock-provisioner-runner
    # oc delete serviceaccounts glusterblock-provisioner
    serviceaccount "glusterblock-provisioner" deleted
    # oc delete clusterrolebindings.authorization.openshift.io glusterblock-provisioner
  19. Execute the following commands to deploy the gluster-block provisioner:

    # sed -e 's/\\\${NAMESPACE}/<NAMESPACE>/' /usr/share/heketi/templates/glusterblock-provisioner.yaml | oc create -f -
    # oc adm policy add-cluster-role-to-user glusterblock-provisioner-runner system:serviceaccount:<NAMESPACE>:glusterblock-provisioner

    For example:

    # sed -e 's/\\\${NAMESPACE}/storage-project/' /usr/share/heketi/templates/glusterblock-provisioner.yaml | oc create -f -
    # oc adm policy add-cluster-role-to-user glusterblock-provisioner-runner system:serviceaccount:storage-project:glusterblock-provisioner
  20. Brick multiplexing is a feature that allows adding multiple bricks into one process. This reduces resource consumption and allows us to run more bricks than before with the same memory consumption. It is enabled by default from Container-Native Storage 3.6 onward. During an upgrade from Container-Native Storage 3.10 to Red Hat Openshift Container Storage 3.11, to turn brick multiplexing on, execute the following commands:

    1. To exec into the Gluster pod, execute the following command and rsh into any of the gluster pods:

      # oc rsh <gluster_pod_name>
    2. Verify the brick multiplex status:

      # gluster v get all all
    3. If it is disabled, then execute the following command to enable brick multiplexing:

      Note

      Ensure that all volumes are in a stop state or no bricks are running while brick multiplexing is enabled.

      # gluster volume set all cluster.brick-multiplex on

      For example:

      # oc rsh glusterfs-registry-g6vd9
      
      sh-4.2# gluster volume set all cluster.brick-multiplex on
      Brick-multiplexing is supported only for container workloads (Independent or Converged mode). Also it is advised to make sure that either all volumes are in stopped state or no bricks are running before this option is modified.Do you still want to continue? (y/n) y
      volume set: success
    4. List all the volumes in the trusted storage pool. This step is only required if the volume set operation is performed:

      For example:

      # gluster volume list
      
      heketidbstorage
      vol_194049d2565d2a4ad78ef0483e04711e
      ...
      ...

      Restart all the volumes. This step is only required if the volume set operation is performed along with the previous step:

      # gluster vol stop <VOLNAME>
      # gluster vol start <VOLNAME>
  21. Support for S3 compatible Object Store in Red Hat Openshift Container Storage is under technology preview. To enable S3 compatible object store, see https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html/operations_guide/s3_object_store.

    Note

    After upgrading the glusterfs registry pods, proceed with the steps listed in ] to bring back your heketi pod and then proceed with the steps listed in xref:chap-upgrade_client_common[ to upgrade the client on Red Hat Openshift Container Platform Nodes.

6.2.4. Upgrading if existing version deployed by using Ansible

6.2.4.1. Upgrading Heketi Server

The following commands must be executed on the client machine.

Note

"yum update cns-deploy -y" is not required to be executed if OCS 3.10 was deployed via Ansible.

  1. Backup the Heketi database file

    # heketi-cli db dump > heketi-db-dump-$(date -I).json
    Note

    The json file created can be used to restore and therefore should be stored in persistent storage of your choice.

  2. Execute the following command to update the heketi client packages. Update the heketi-client package on all the OCP nodes where it is installed. Newer installations may not have the heketi-client rpm installed on any OCP nodes:

    # yum update heketi-client -y
  3. Execute the following command to get the current HEKETI_ADMIN_KEY.

    The OCS admin can choose to set any phrase for user key as long as it is not used by their infrastructure. It is not used by any of the OCS default installed resources.

    # oc get secret heketi-registry-admin-secret -o jsonpath='{.data.key}'|base64 -d;echo
  4. If the HEKETI_USER_KEY was set previously, you can obtain it by using the following command:

    # oc describe pod <heketi-pod>
  5. Execute the following step to edit the template:

    1. If the existing template has IMAGE_NAME, then edit the template to change the HEKETI_USER_KEY, HEKETI_ADMIN_KEY, HEKETI_ROUTE, IMAGE_NAME, CLUSTER_NAME and HEKETI_LVM_WRAPPER as shown in the example below.

      # oc edit template heketi
      parameters:
      - description: Set secret for those creating volumes as type user
        displayName: Heketi User Secret
        name: HEKETI_USER_KEY
        value: <heketiuserkey>
      - description: Set secret for administration of the Heketi service as user admin
        displayName: Heketi Administrator Secret
        name: HEKETI_ADMIN_KEY
        value: <adminkey>
      - description: Set the executor type, kubernetes or ssh
        displayName: heketi executor type
        name: HEKETI_EXECUTOR
        value: kubernetes
      - description: Set the hostname for the route URL
        displayName: heketi route name
        name: HEKETI_ROUTE
        value: heketi-registry
      - displayName: heketi container image name
        name: IMAGE_NAME
        required: true
        value: registry.redhat.io/rhgs3/rhgs-volmanager-rhel7:v3.11.6
      - description: A unique name to identify this heketi service, useful for running
        multiple heketi instances
        displayName: GlusterFS cluster name
        name: CLUSTER_NAME
        value: registry
      - description: Heketi can use a wrapper to execute LVM commands, i.e. run commands in the host namespace instead of in the Gluster container
        name: HEKETI_LVM_WRAPPER
        displayName: Wrapper for executing LVM commands
        value: /usr/sbin/exec-on-host
    2. If the existing template has IMAGE_NAME and IMAGE_VERSION as two parameters, then edit the template to change the HEKETI_USER_KEY, HEKETI_ADMIN_KEY, HEKETI_ROUTE, IMAGE_NAME, IMAGE_VERSION, CLUSTER_NAME and HEKETI_LVM_WRAPPER as shown in the example below.

      # oc edit template heketi
      parameters:
      - description: Set secret for those creating volumes as type user
        displayName: Heketi User Secret
        name: HEKETI_USER_KEY
        value: <heketiuserkey>
      - description: Set secret for administration of the Heketi service as user admin
        displayName: Heketi Administrator Secret
        name: HEKETI_ADMIN_KEY
        value: <adminkey>
      - description: Set the executor type, kubernetes or ssh
        displayName: heketi executor type
        name: HEKETI_EXECUTOR
        value: kubernetes
      - description: Set the hostname for the route URL
        displayName: heketi route name
        name: HEKETI_ROUTE
        value: heketi-registry
      - displayName: heketi container image name
        name: IMAGE_NAME
        required: true
        value: registry.redhat.io/rhgs3/rhgs-volmanager-rhel7
      - displayName: heketi container image version
        name: IMAGE_VERSION
        required: true
        value: v3.11.6
      - description: A unique name to identify this heketi service, useful for running multiple heketi instances
        displayName: GlusterFS-registry cluster name
        name: CLUSTER_NAME
        value: registry
      - description: Heketi can use a wrapper to execute LVM commands, i.e. run commands in the host namespace instead of in the Gluster container
        name: HEKETI_LVM_WRAPPER
        displayName: Wrapper for executing LVM commands
        value: /usr/sbin/exec-on-host
      Note

      If a cluster has more than 1000 volumes refer to How to change the default PVS limit in Openshift Container Storage and add the required parameters before proceeding with the upgrade.

  6. Execute the following command to delete the deployment configuration, service, and route for heketi:

    # oc delete deploymentconfig,service,route heketi-registry
  7. Execute the following command to deploy the Heketi service, route, and deployment configuration which will be used to create persistent volumes for OpenShift:

    # oc process heketi | oc create -f -
    
    service "heketi-registry" created
    route "heketi-registry" created
    deploymentconfig-registry "heketi" created
    Note

    It is recommended that the heketidbstorage volume be tuned for db workloads. Newly installed Openshift Container Storage deployments tune the heketidbstorage volume automatically. For older deployments, follow the KCS article Planning to run containerized DB or nosql workloads on Openshift Container Storage? and perform the volume set operation for the volume heketidbstorage.

  8. Execute the following command to verify that the containers are running:

    # oc get pods

    For example:

    # oc get pods
    NAME                                           READY     STATUS    RESTARTS   AGE
    glusterblock-registry-provisioner-dc-1-lm7ht   1/1       Running   81         14d
    glusterfs-registry-l25b9                       1/1       Running   10         14d
    glusterfs-registry-vl6qs                       1/1       Running   10         14d
    glusterfs-registry-zhxwg                       1/1       Running   10         14d
    heketi-registry-1-54bwf                        1/1       Running   10         14d

6.2.4.2. Upgrading the Red Hat Gluster Storage Registry Pods

The following commands must be executed on the client machine.

Following are the steps for updating a DaemonSet for glusterfs:

  1. Execute the following steps to stop the Heketi pod to prevent it from accepting any new request for volume creation or volume deletion:

    1. Execute the following command to access your project:

      # oc project <project_name>

      For example:

      # oc project storage-project
    2. Execute the following command to get the DeploymentConfig:

      # oc get dc
    3. Execute the following command to set heketi server to accept requests only from the local-client:

      # heketi-cli server mode set local-client
    4. Wait for the ongoing operations to complete and execute the following command to monitor if there are any ongoing operations:

      # heketi-cli server operations info
    5. Execute the following command to reduce the replica count from 1 to 0. This brings down the Heketi pod:

      # oc scale dc <heketi_dc> --replicas=0
    6. Execute the following command to verify that the heketi pod is no longer present:

      # oc get pods
  2. Execute the following command to find the DaemonSet name for gluster

    # oc get ds
  3. Execute the following command to delete the DaemonSet:

    # oc delete ds <ds-name> --cascade=false

    Using --cascade=false option while deleting the old DaemonSet does not delete the glusterfs_registry pods but deletes only the DaemonSet. After deleting the old DaemonSet, you must load the new one. When you manually delete the old pods, the new pods which are created will have the configurations of the new DaemonSet.

    For example,

    # oc delete ds glusterfs-registry --cascade=false
    daemonset "glusterfs-registry" deleted
  4. Execute the following commands to verify all the old pods are up:

    # oc get pods

    For example,

    # oc get pods
    NAME                                           READY     STATUS    RESTARTS   AGE
    glusterblock-registry-provisioner-dc-1-nvnhc   1/1       Running   1          7d
    glusterfs-registry-4cpcc                       1/1       Running   0          7d
    glusterfs-registry-9xj78                       1/1       Running   0          7d
    glusterfs-registry-b9p5j                       1/1       Running   0          7d
  5. Execute the following command to delete the old glusterfs template.

     # oc delete templates glusterfs
  6. Execute the following command to register new glusterfs template.

    # oc create -f /usr/share/ansible/openshift-ansible/roles/openshift_storage_glusterfs/files/glusterfs-template.yml
    template "glusterfs" created
  7. Execute the following command to edit the old glusterfs template.

    1. If the template has IMAGE_NAME, then update the glusterfs template as following. For example:

      # oc edit template glusterfs
      
      - description: Labels which define the daemonset node selector. Must contain at least
          one label of the format \'glusterfs=<CLUSTER_NAME>-host\'
        displayName: Daemonset Node Labels
        name: NODE_LABELS
        value: '{ "glusterfs": "registry-host" }'
      - displayName: GlusterFS container image name
        name: IMAGE_NAME
        required: true
        value: registry.redhat.io/rhgs3/rhgs-server-rhel7:v3.11.6
      - description: A unique name to identify which heketi service manages this cluster,
          useful for running multiple heketi instances
        displayName: GlusterFS cluster name
        name: CLUSTER_NAME
        value: registry
    2. If the template has IMAGE_NAME and IMAGE_VERSION as two separate parameters, then update the glusterfs template as following. For example:

      # oc edit template glusterfs
      - description: Labels which define the daemonset node selector. Must contain at least one label of the format \'glusterfs=<CLUSTER_NAME>-host\'
      displayName: Daemonset Node Labels
      name: NODE_LABELS
      value: '{ "glusterfs": "registry-host" }'
      - displayName: GlusterFS container image name
      name: IMAGE_NAME
      required: true
      value: registry.redhat.io/rhgs3/rhgs-server-rhel7
      - description: A unique name to identify which heketi service manages this cluster,
      useful for running multiple heketi instances
      - displayName: GlusterFS container image version
      name: IMAGE_VERSION
      required: true
      value: v3.11.6
      - displayName: GlusterFS cluster name
      name: CLUSTER_NAME
      value: registry
      Note
  8. Label all the OpenShift Container Platform nodes that has the Red Hat Gluster Storage pods:

    1. Check if the nodes are labelled with the appropriate label by using the following command:

      # oc get nodes -l glusterfs=registry-host
  9. Execute the following commands to create the gluster DaemonSet:

    # oc process glusterfs | oc create -f -

    For example,

    # oc process glusterfs | oc create -f -
    Deamonset “glusterfs-registry” created
  10. Execute the following command to identify the old glusterfs_registry pods that needs to be deleted:

    # oc get pods

    For example,

    # oc get pods
    NAME                                           READY     STATUS    RESTARTS   AGE
    glusterblock-registry-provisioner-dc-1-nvnhc   1/1       Running   1          7d
    glusterfs-registry-4cpcc                       1/1       Running   0          7d
    glusterfs-registry-9xj78                       1/1       Running   0          7d
    glusterfs-registry-b9p5j                       1/1       Running   0          7d
  11. Execute the following command and ensure that the bricks are not more than 90% full:

    # df -kh | grep -v ^Filesystem | awk '{if(int($5)>90) print $0}'
    Note

    If the bricks are close to 100% utilization, then the Logical Volume Manager(LVM) activation for these bricks may take a long time or can get stuck once the pod or node is rebooted. It is advised to bring down the utilization of that brick or expand the physical volume(PV) that is using the logical volume(LV).

  12. Execute the following command to delete the old glusterfs-registry pods. glusterfs-registry pods should follow rolling upgrade. Hence, you must ensure that the new pod is running before deleting the next old glusterfs-registry pods. We support OnDelete Strategy DaemonSet update strategy. With OnDelete Strategy update strategy, after you update a DaemonSet template, new DaemonSet pods will only be created when you manually delete old DaemonSet pods.

    1. To delete the old glusterfs-registry pods, execute the following command:

      # oc delete pod <gluster_pod>

      For example,

      # oc delete pod glusterfs-registry-4cpcc
      pod “glusterfs-registry-4cpcc” deleted
      Note

      Before deleting the next pod, self heal check has to be made:

      1. Run the following command to access shell on glusterfs-registry pods:

        # oc rsh <gluster_pod_name>
      2. Run the following command to check the self-heal status of all the volumes: :

        # for each_volume in `gluster volume list`; do gluster volume heal $each_volume info ; done | grep "Number of entries: [^0]$"
    2. The delete pod command will terminate the old pod and create a new pod. Run # oc get pods -w and check the Age of the pod and READY status should be 1/1. The following is the example output showing the status progression from termination to creation of the pod.

      # oc get pods -w
      NAME                             READY     STATUS        RESTARTS   AGE
      glusterfs-registry-4cpcc                  1/1       Terminating   0          3d
      …
  13. Execute the following command to verify that the pods are running:

    # oc get pods

    For example,

    # oc get pods
    NAME                                           READY     STATUS    RESTARTS   AGE
    glusterblock-registry-provisioner-dc-1-nvnhc   1/1       Running   1          7d
    glusterfs-registry-abmqa                       1/1       Running   0          01m
    glusterfs-registry-9xj78                       1/1       Running   0          7d
    glusterfs-registry-b9p5j                       1/1       Running   0          7d
  14. Execute the following commands to verify if you have upgraded the pod to the latest version:

    # oc rsh <gluster_registry_pod_name> glusterd --version

    For example:

    # oc rsh glusterfs-registry-abmqa glusterd --version
    glusterfs 6.0
    # rpm -qa|grep gluster
  15. Check the Red Hat Gluster Storage op-version by executing the following command on one of the glusterfs-registry pods.

    # gluster vol get all cluster.op-version
  16. After you upgrade the Gluster pods, ensure that you set Heketi back to operational mode:

    • Scale up the DC (Deployment Configuration).

      # oc scale dc <heketi_dc> --replicas=1
  17. Set the cluster.op-version to 70000 on any one of the pods:

    Note

    Ensure all the glusterfs-registry pods are updated before changing the cluster.op-version.

    # gluster volume set all cluster.op-version 70000
  18. Execute the following steps to enable server.tcp-user-timeout on all volumes.

    Note

    The "server.tcp-user-timeout" option specifies the maximum amount of the time (in seconds) the transmitted data from the application can remain unacknowledged from the brick.

    It is used to detect force disconnections and dead connections (if a node dies unexpectedly, a firewall is activated, etc.,) early and make it possible for applications to reduce the overall failover time.

    1. List the glusterfs pod using the following command:

      # oc get pods

      For example:

      # oc get pods
      NAME                                           READY     STATUS    RESTARTS   AGE
      glusterblock-registry-provisioner-dc-1-nvnhc   1/1       Running   1          7d
      glusterfs-registry-3cpqt                       1/1       Running   0          01m
      glusterfs-registry-4xj33                       1/1       Running   0          10m
      glusterfs-registry-l9p2p                       1/1       Running   0          20m
    2. Remote shell into one of the glusterfs-registry pods. For example:

      # oc rsh glusterfs-registry-g6vd9
    3. Execute the following command:

      # for eachVolume in `gluster volume list`; do echo $eachVolume; gluster volume set $eachVolume server.tcp-user-timeout 42 ; done

      For example:

      # for eachVolume in `gluster volume list`; do echo $eachVolume; gluster volume set $eachVolume server.tcp-user-timeout 42 ; done
      volume1
      volume set: success
      volume2
      volume set: success
  19. If a gluster-block-registry-provisoner-pod already exists then delete it by executing the following commands:

    # oc delete dc <gluster-block-registry-dc>

    For example:

    # oc delete dc glusterblock-registry-provisioner-dc
  20. Execute the following command to delete the old glusterblock provisioner template.

     # oc delete templates glusterblock-provisioner
  21. Execute the following command to register new glusterblock provisioner template, see Templates on GitHub. Copy and paste to new-block-prov.yaml.
    For example,

    # oc create -f new-block-prov.yaml
    template.template.openshift.io/glusterblock-provisioner created
  22. Depending on the OCP version, edit the glusterblock-provisioner template to change the IMAGE_NAME and NAMESPACE.

    # oc edit template glusterblock-provisioner
    - displayName: glusterblock provisioner container image name
      name: IMAGE_NAME
      required: true
      value: registry.redhat.io/rhgs3/rhgs-gluster-block-prov-rhel7:v3.11.6
    - description: The namespace in which these resources are being created
      displayName: glusterblock provisioner namespace
      name: NAMESPACE
      required: true
      value: glusterfs-registry
    - description: A unique name to identify which heketi service manages this cluster, useful for running multiple heketi instances
      displayName: GlusterFS cluster name
      name: CLUSTER_NAME
      value: registry
    • If the template has IMAGE_NAME and IMAGE_VERSION as two separate parameters, then update the glusterblock-provisioner template as following.
      For example:

      # oc edit template glusterblock-provisioner
      
      - displayName: glusterblock provisioner container image name
        name: IMAGE_NAME
        required: true
        value: registry.redhat.io/rhgs3/rhgs-gluster-block-prov-rhel7
      - displayName: glusterblock provisioner container image version
        name: IMAGE_VERSION
        required: true
        value: v3.11.6
      - description: The namespace in which these resources are being created
        displayName: glusterblock provisioner namespace
        name: NAMESPACE
        required: true
        value: glusterfs-registry
      - description: A unique name to identify which heketi service manages this cluster,
        useful for running multiple heketi instances
        displayName: GlusterFS cluster name
        name: CLUSTER_NAME
        value: registry
  23. Delete the following resources from the old pod

    # oc delete clusterroles.authorization.openshift.io glusterblock-provisioner-runner
    # oc delete serviceaccounts glusterblock-registry-provisioner
    # oc delete clusterrolebindings.authorization.openshift.io glusterblock-registry-provisioner
  24. Before running oc process determine the correct provisioner name. If there are more than one gluster block provisioner running in your cluster the names must differ from all other provisioners.
    For example,

    • If there are 2 or more provisioners the name should be gluster.org/glusterblock-<namespace> where, namespace is replaced by the namespace that the provisioner is deployed in.
    • If there is only one provisioner, installed prior to 3.11.5, gluster.org/glusterblock is sufficent. If the name currently in use already has a unique namespace suffix, reuse the existing name.
  25. After editing the template, execute the following command to create the deployment configuration:

    # oc process -p PROVISIONER_NAME=<provisioner-name> glusterblock-provisioner -o yaml | oc create -f -

    For example:

     # oc process -p PROVISIONER_NAME=gluster.org/glusterblock-infra-storage glusterblock-provisioner -o yaml | oc create -f -
      clusterrole.authorization.openshift.io/glusterblock-provisioner-runner created
      serviceaccount/glusterblock-registry-provisioner created
      clusterrolebinding.authorization.openshift.io/glusterblock-registry-provisioner created
      deploymentconfig.apps.openshift.io/glusterblock-registry-provisioner-dc created
  26. Brick multiplexing is a feature that allows adding multiple bricks into one process. This reduces resource consumption and allows us to run more bricks than before with the same memory consumption. It is enabled by default from Container-Native Storage 3.6 onward. During an upgrade from Container-Native Storage 3.10 to Red Hat Openshift Container Storage 3.11, to turn brick multiplexing on, execute the following commands:

    1. To exec into the Gluster pod, execute the following command and rsh into any of the gluster pods:

      # oc rsh <gluster_pod_name>
    2. Verify the brick multiplex status:

      # gluster v get all all
    3. If it is disabled, then execute the following command to enable brick multiplexing:

      Note

      Ensure that all volumes are in a stop state or no bricks are running while brick multiplexing is enabled.

      # gluster volume set all cluster.brick-multiplex on

      For example:

      # oc rsh glusterfs-registry-g6vd9
      
      sh-4.2# gluster volume set all cluster.brick-multiplex on
      Brick-multiplexing is supported only for container workloads (Independent or Converged mode). Also it is advised to make sure that either all volumes are in stopped state or no bricks are running before this option is modified.Do you still want to continue? (y/n) y
      volume set: success
    4. List all the volumes in the trusted storage pool. This step is only required if the volume set operation is performed:

      For example:

      # gluster volume list
      
      heketidbstorage
      vol_194049d2565d2a4ad78ef0483e04711e
      ...
      ...

      Restart all the volumes. This step is only required if the volume set operation is performed along with the previous step:

      # gluster vol stop <VOLNAME>
      # gluster vol start <VOLNAME>
  27. Support for S3 compatible Object Store in Red Hat Openshift Container Storage is under technology preview. To enable S3 compatible object store, see https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html/operations_guide/s3_object_store.

    Note

    After upgrading the glusterfs registry pods, proceed with the steps listed in ] to bring back your heketi pod and then proceed with the steps listed in xref:chap-upgrade_client_common[ to upgrade the client on Red Hat Openshift Container Platform Nodes.

  28. All storage classes that use gluster block volume provisioning must match exactly to one of the provisioner names in the cluster. To check the list of storage classes that refer to a block provisioner, in a given namespace, run the following command:

    # oc get sc -o custom-columns=NAME:.metadata.name,PROV:.provisioner,RSNS:.parameters.restsecretnamespace | grep 'gluster.org/glusterblock' | grep <namespace>

    Example:

    # oc get sc -o custom-columns=NAME:.metadata.name,PROV:.provisioner,RSNS:.parameters.restsecretnamespace | grep 'gluster.org/glusterblock' | grep infra-storage
      glusterfs-registry-block   gluster.org/glusterblock               infra-storage

    Check each storage class provisioner name, if it does not match the block provisioner name configured for that namespace it must be updated. If the block provisioner name already matches the configured provisioner name, nothing else needs to be done. Use the list generated above and include all storage class names where the provionser name must be updated.
    For every storage class in this list do the following:

    # oc get sc  -o yaml <storageclass>  > storageclass-to-edit.yaml
    # oc delete sc  <storageclass>
    # sed 's,gluster.org/glusterblock$,gluster.org/glusterblock-<namespace>,' storageclass-to-edit.yaml | oc create -f -

    Example:

    # oc get sc -o yaml glusterfs-registry-block > storageclass-to-edit.yaml
    # oc delete sc glusterfs-registry-block
    storageclass.storage.k8s.io "glusterfs-registry-block" deleted
    # sed 's,gluster.org/glusterblock$,gluster.org/glusterblock-infra-storage,' storageclass-to-edit.yaml | oc create -f -
    storageclass.storage.k8s.io/glusterfs-registry-block created

6.3. Starting the Heketi Pods

Execute the following commands on the client machine for both glusterfs and registry namespace.

  1. Execute the following command to navigate to the project where the Heketi pods are running:

    # oc project <project_name>

    For example for glusterfs namespace:

    # oc project glusterfs

    For example for registry namespace:

    # oc project glusterfs-registry
  2. Execute the following command to get the DeploymentConfig:

    # oc get dc

    For example, on a glusterfs-registry project:

    # oc get dc
    NAME                                   REVISION   DESIRED   CURRENT   TRIGGERED BY
    glusterblock-registry-provisioner-dc   1          1         1         config
    heketi-registry                        1          1         1         config

    For example, on a glusterfs project:

    # oc get dc
    NAME                                  REVISION   DESIRED   CURRENT   TRIGGERED BY
    glusterblock-storage-provisioner-dc   1          1         1         config
    heketi-storage                        1          1         1         config
  3. Execute the following command to increase the replica count from 0 to 1. This brings back the Heketi pod:

    # oc scale dc <heketi_dc> --replicas=1
  4. Execute the following command to verify that the heketi pod is present in both glusterfs and glusterfs-registry namespace:

    # oc get pods

    For example for glusterfs:

    # oc get pods
    glusterblock-storage-provisioner-dc-1-fc8sc   1/1       Running       0          3d
    glusterfs-storage-bd6kv                       1/1       Running       0          2h
    glusterfs-storage-vhpcw                       1/1       Running       1          1d
    glusterfs-storage-z6nkk                       1/1       Running       0          1d
    heketi-storage-1-6sccl                        1/1       Running       0          2h

    For example for registry pods:

    # oc get pods
    NAME                                           READY     STATUS    RESTARTS   AGE
    glusterblock-registry-provisioner-dc-1-c59rn   1/1       Running   0          2d
    glusterfs-registry-987g6                       1/1       Running   4          2m
    glusterfs-registry-sd7z4                       1/1       Running   4          2m
    glusterfs-registry-tlzfb                       1/1       Running   4          2m
    
    heketi-registry-1-qktss                        1/1       Running   0          1d

6.4. Upgrading the client on Red Hat OpenShift Container Platform nodes

Execute the following commands on each of the nodes:

  1. To drain the pod, execute the following command on the master node (or any node with cluster-admin access):

    # oc adm drain <node_name> --ignore-daemonsets
  2. To check if all the pods are drained, execute the following command on the master node (or any node with cluster-admin access) :

    # oc get pods --all-namespaces --field-selector=spec.nodeName=<node_name>
  3. Execute the following command to upgrade the client node to the latest glusterfs-fuse version:

    # yum update glusterfs-fuse
  4. To enable node for pod scheduling execute the following command on the master node (or any node with cluster-admin access):

    # oc adm manage-node --schedulable=true <node_name>
  5. Create and add the following content to the multipath.conf file:

    Note

    Make sure that the changes to multipath.conf and reloading of multipathd are done only after all the server nodes are upgraded.

    # cat >> /etc/multipath.conf <<EOF
    # LIO iSCSI
    devices {
      device {
        vendor "LIO-ORG"
        user_friendly_names "yes" # names like mpatha
        path_grouping_policy "failover" # one path per group
        hardware_handler "1 alua"
        path_selector "round-robin 0"
        failback immediate
        path_checker "tur"
        prio "alua"
        no_path_retry 120
        rr_weight "uniform"
      }
    }
    EOF
  6. Execute the following commands to start multipath daemon and [re]load the multipath configuration:

    # systemctl start multipathd
    # systemctl reload multipathd