-
Language:
English
-
Language:
English
7.2. Upgrading your Independent Mode Setup
Follow the steps in the sections ahead to upgrade your independent mode Setup.
7.2.1. Upgrading the Red Hat Gluster Storage Cluster
To upgrade the Red Hat Gluster Storage cluster, see In-Service Software Upgrade.
7.2.2. Upgrading/Migration of Heketi in RHGS node
Note
If Heketi is in an Openshift node, then skip this section and see Section 7.2.4.1, “Upgrading Heketi in Openshift node” instead.
Important
- In OCS 3.11, upgrade of Heketi in RHGS node is not supported. Hence, you have to migrate heketi to a new heketi pod.
- Ensure to migrate to the supported heketi deployment now, as there might not be a migration path in the future versions.
- Ensure that cns-deploy rpm is installed in the master node. This provides template files necessary to setup heketi pod.
# subscription-manager repos --enable=rh-gluster-3-for-rhel-7-server-rpms
# yum install cns-deploy
- Use the newly created containerized Red Hat Gluster Storage project on the master node:
# oc project <project-name>
For example:# oc project gluster
- Execute the following command on the master node to create the service account:
# oc create -f /usr/share/heketi/templates/heketi-service-account.yaml serviceaccount/heketi-service-account created
- Execute the following command on the master node to install the heketi template:
# oc create -f /usr/share/heketi/templates/heketi-template.yaml template.template.openshift.io/heketi created
- Verify if the templates are created
# oc get templates NAME DESCRIPTION PARAMETERS OBJECTS heketi Heketi service deployment template 5 (3 blank) 3
- Execute the following command on the master node to grant the heketi Service Account the necessary privileges:
# oc policy add-role-to-user edit system:serviceaccount:gluster:heketi-service-account role "edit" added: "system:serviceaccount:gluster:heketi-service-account"
# oc adm policy add-scc-to-user privileged -z heketi-service-account scc "privileged" added to: ["system:serviceaccount:gluster:heketi-service-account"]
- On the RHGS node, where heketi is running, execute the following commands:
- Create the heketidbstorage volume:
# heketi-cli volume create --size=2 --name=heketidbstorage
- Mount the volume:
# mount -t glusterfs 192.168.11.192:heketidbstorage /mnt/
where 192.168.11.192 is one of the RHGS node. - Stop the heketi service:
# systemctl stop heketi
- Disable the heketi service:
# systemctl disable heketi
- Copy the heketi db to the heketidbstorage volume:
# cp /var/lib/heketi/heketi.db /mnt/
- Unmount the volume:
# umount /mnt
- Copy the following files from the heketi node to the master node:
# scp /etc/heketi/heketi.json topology.json /etc/heketi/heketi_key OCP_master_node:/root/
where OCP_master_node is the hostname of the master node.
- On the master node, set the environment variables for the following three files that were copied from the heketi node. Add the following lines to ~/.bashrc file and run the bash command to apply and save the changes:
export SSH_KEYFILE=heketi_key export TOPOLOGY=topology.json export HEKETI_CONFIG=heketi.json
Note
If you have changed the value for "keyfile" in /etc/heketi/heketi.json to a different value, change here accordingly. - Execute the following command to create a secret to hold the configuration file:
# oc create secret generic heketi-config-secret --from-file=${SSH_KEYFILE} --from-file=${HEKETI_CONFIG} --from-file=${TOPOLOGY} secret/heketi-config-secret created - Execute the following command to label the secret:
# oc label --overwrite secret heketi-config-secret glusterfs=heketi-config-secret heketi=config-secret secret/heketi-config-secret labeled
- Get the IP addresses of all the glusterfs nodes, from the heketi-gluster-endpoints.yml file. For example:
# cat heketi-gluster-endpoints.yaml apiVersion: v1 kind: Endpoints metadata: name: heketi-storage-endpoints subsets: - addresses: - ip: 192.168.11.208 ports: - port: 1 - addresses: - ip: 192.168.11.176 ports: - port: 1 - addresses: - ip: 192.168.11.192 ports: - port: 1
In the above example, 192.168.11.208, 192.168.11.176, 192.168.11.192 are the glusterfs nodes. - Execute the following command to create the endpoints:
# oc create -f ./heketi-gluster-endpoints.yaml
# cat heketi-gluster-service.yaml apiVersion: v1 kind: Service metadata: name: heketi-storage-endpoints spec: ports: - port: 1
- Execute the following command to create the service:
# oc create -f ./heketi-gluster-service.yaml
- Execute the following command to deploy the Heketi service, route, and deployment configuration which will be used to create persistent volumes for OpenShift:
# # oc process heketi | oc create -f - service/heketi created route.route.openshift.io/heketi created deploymentconfig.apps.openshift.io/heketi created
- To verify if Heketi is migrated execute the following command on the master node:
# oc rsh po/<heketi-pod-name>
For example:# oc rsh po/heketi-1-p65c6
- Execute the following command to check the cluster IDs
# heketi-cli cluster list
From the output verify if the cluster ID matches with the old cluster.
7.2.3. Upgrading if existing version deployed using cns-deploy
7.2.3.1. Upgrading Heketi in Openshift node
The following commands must be executed on the client machine.
- Execute the following command to update the heketi client and cns-deploy packages:
# yum update cns-deploy -y # yum update heketi-client -y
- Backup the Heketi database file
# oc rsh <heketi_pod_name> # cp -a /var/lib/heketi/heketi.db /var/lib/heketi/heketi.db.`date +%s`.`heketi --version | awk '{print $2}'` # exit - Execute the following command to delete the heketi template.
# oc delete templates heketi
- Execute the following command to get the current HEKETI_ADMIN_KEY.The OCS admin can choose to set any phrase for user key as long as it is not used by their infrastructure. It is not used by any of the OCS default installed resources.
oc get secret heketi-storage-admin-secret -o jsonpath='{.data.key}'|base64 -d;echo - Execute the following command to install the heketi template.
# oc create -f /usr/share/heketi/templates/heketi-template.yaml template "heketi" created
- Execute the following command to grant the heketi Service Account the necessary privileges.
# oc policy add-role-to-user edit system:serviceaccount:<project_name>:heketi-service-account # oc adm policy add-scc-to-user privileged -z heketi-service-account
For example,# oc policy add-role-to-user edit system:serviceaccount:storage-project:heketi-service-account # oc adm policy add-scc-to-user privileged -z heketi-service-account
- Execute the following command to generate a new heketi configuration file.
# sed -e "s/\${HEKETI_EXECUTOR}/ssh/" -e "s#\${HEKETI_FSTAB}#/etc/fstab#" -e "s/\${SSH_PORT}/22/" -e "s/\${SSH_USER}/root/" -e "s/\${SSH_SUDO}/false/" -e "s/\${BLOCK_HOST_CREATE}/true/" -e "s/\${BLOCK_HOST_SIZE}/500/" "/usr/share/heketi/templates/heketi.json.template" > heketi.json- The
BLOCK_HOST_SIZEparameter controls the size (in GB) of the automatically created Red Hat Gluster Storage volumes hosting the gluster-block volumes (For more information, see https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html-single/operations_guide/#Block_Storage). This default configuration will dynamically create block-hosting volumes of 500GB in size as more space is required. - Alternatively, copy the file
/usr/share/heketi/templates/heketi.json.templatetoheketi.jsonin the current directory and edit the new file directly, replacing each "${VARIABLE}" string with the required parameter.Note
JSON formatting is strictly required (e.g. no trailing spaces, booleans in all lowercase).
Note
If theheketi-config-secretfile already exists, then delete the file and run the following command.Execute the following command to create a secret to hold the configuration file.# oc create secret generic heketi-config-secret --from-file=private_key=${SSH_KEYFILE} --from-file=./heketi.json- Execute the following command to delete the deployment configuration, service, and route for heketi:
# oc delete deploymentconfig,service,route heketi
- Execute the following command to edit the heketi template. Edit the HEKETI_USER_KEY, HEKETI_ADMIN_KEY, and HEKETI_EXECUTOR parameters.
# oc edit template heketi parameters: - description: Set secret for those creating volumes as type _user_ displayName: Heketi User Secret name: HEKETI_USER_KEY value: <heketiuserkey> - description: Set secret for administration of the Heketi service as user _admin_ displayName: Heketi Administrator Secret name: HEKETI_ADMIN_KEY value: <adminkey> - description: Set the executor type, kubernetes or ssh displayName: heketi executor type name: HEKETI_EXECUTOR value: ssh - description: Set the fstab path, file that is populated with bricks that heketi creates displayName: heketi fstab path name: HEKETI_FSTAB value: /etc/fstab - description: Set the hostname for the route URL displayName: heketi route name name: HEKETI_ROUTE value: heketi-storage - displayName: heketi container image name name: IMAGE_NAME required: true value: rhgs3/rhgs-volmanager-rhel7:v3.10 - description: A unique name to identify this heketi service, useful for running multiple heketi instances displayName: GlusterFS cluster name name: CLUSTER_NAME value: storage - Execute the following command to deploy the Heketi service, route, and deployment configuration which will be used to create persistent volumes for OpenShift:
# oc process heketi | oc create -f - service "heketi" created route "heketi" created deploymentconfig "heketi" created
- Execute the following command to verify that the containers are running:
# oc get pods
For example:# oc get pods NAME READY STATUS RESTARTS AGE glusterfs-0h68l 1/1 Running 0 3d glusterfs-0vcf3 1/1 Running 0 3d glusterfs-gr9gh 1/1 Running 0 3d heketi-1-zpw4d 1/1 Running 0 3h storage-project-router-2-db2wl 1/1 Running 0 4d
7.2.3.2. Upgrading Gluster Block
Execute the following steps to upgrade gluster block.
- Execute the following command to upgrade the gluster block:
# yum update gluster-block
- Enable and start the gluster block service:
# systemctl enable gluster-blockd # systemctl start gluster-blockd
- Execute the following command to update the heketi client and cns-deploy packages
# yum update cns-deploy -y # yum update heketi-client -y
- To use gluster block, add the following two parameters to the
glusterfssection in the heketi configuration file at /etc/heketi/heketi.JSON:auto_create_block_hosting_volume block_hosting_volume_size
Where:auto_create_block_hosting_volume: Creates Block Hosting volumes automatically if not found or if the existing volume is exhausted. To enable this, set the value totrue.block_hosting_volume_size: New block hosting volume will be created in the size mentioned. This is considered only if auto_create_block_hosting_volume is set to true. Recommended size is 500G.For example:..... ..... "glusterfs" : { "executor" : "ssh", "db" : "/var/lib/heketi/heketi.db", "sshexec" : { "rebalance_on_expansion": true, "keyfile" : "/etc/heketi/private_key" }, "auto_create_block_hosting_volume": true, "block_hosting_volume_size": 500G }, ..... ..... - Restart the Heketi service:
# systemctl restart heketi
Note
This step is not applicable if heketi is running as a pod in the Openshift cluster. - If a gluster-block-provisoner-pod already exists then delete it by executing the following commands:
# oc delete dc <gluster-block-dc>
For example:# oc delete dc glusterblock-provisioner-dc
- Execute the following commands to deploy the gluster-block provisioner:
# sed -e 's/\\\${NAMESPACE}/<NAMESPACE>/' /usr/share/heketi/templates/glusterblock-provisioner.yaml | oc create -f -# oc adm policy add-cluster-role-to-user glusterblock-provisioner-runner system:serviceaccount:<NAMESPACE>:glusterblock-provisioner
For example:# sed -e 's/\\\${NAMESPACE}/storage-project/' /usr/share/heketi/templates/glusterblock-provisioner.yaml | oc create -f -# oc adm policy add-cluster-role-to-user glusterblock-provisioner-runner system:serviceaccount:storage-project:glusterblock-provisioner
- Delete the following resources from the old pod
# oc delete clusterroles.authorization.openshift.io glusterblock-provisioner-runner
# oc delete serviceaccounts glusterblock-registry-provisioner
- Execute the following command to create a glusterblock-provisioner.
# oc process <gluster_block_provisioner_template> | oc create -f -
7.2.4. Upgrading if existing version deployed using Ansible
7.2.4.1. Upgrading Heketi in Openshift node
The following commands must be executed on the client machine.
- Execute the following command to update the heketi client:
# yum update heketi-client -y
- Backup the Heketi database file
# oc rsh <heketi_pod_name> # cp -a /var/lib/heketi/heketi.db /var/lib/heketi/heketi.db.`date +%s`.`heketi --version | awk '{print $2}'` # exit - Execute the following command to get the current HEKETI_ADMIN_KEY.The OCS admin can choose to set any phrase for user key as long as it is not used by their infrastructure. It is not used by any of the OCS default installed resources.
oc get secret heketi-storage-admin-secret -o jsonpath='{.data.key}'|base64 -d;echo - Execute the following step to edit the template:
# oc get templates NAME DESCRIPTION PARAMETERS OBJECTS glusterblock-provisioner glusterblock provisioner 3 (2 blank) 4 template glusterfs GlusterFS DaemonSet 5 (1 blank) 1 template heketi Heketi service deployment 7 (3 blank) 3 template
If the existing template has IMAGE_NAME and IMAGE_VERSION as two parameters, then edit the template to change the HEKETI_EXECUTOR, HEKETI_FSTAB, HEKETI_ROUTE, IMAGE_NAME, IMAGE_VERSION and CLUSTER_NAME as shown in the example below.# oc edit template heketi - description: Set the executor type, kubernetes or ssh displayName: heketi executor type name: HEKETI_EXECUTOR value: ssh - description: Set the fstab path, file that is populated with bricks that heketi creates displayName: heketi fstab path name: HEKETI_FSTAB value: /etc/fstab - description: Set the hostname for the route URL displayName: heketi route name name: HEKETI_ROUTE value: heketi-storage - displayName: heketi container image name name: IMAGE_NAME required: true value: rhgs3/rhgs-volmanager-rhel7 - displayName: heketi container image version name: IMAGE_VERSION required: true value: v3.10 - description: A unique name to identify this heketi service, useful for running multiple heketi instances displayName: GlusterFS cluster name name: CLUSTER_NAME value: storage
If the template has only IMAGE_NAME, then edit the template to change the HEKETI_EXECUTOR, HEKETI_FSTAB, HEKETI_ROUTE, IMAGE_NAME, and CLUSTER_NAME as shown in the example below.- description: Set the executor type, kubernetes or ssh displayName: heketi executor type name: HEKETI_EXECUTOR value: ssh - description: Set the fstab path, file that is populated with bricks that heketi creates displayName: heketi fstab path name: HEKETI_FSTAB value: /etc/fstab - description: Set the hostname for the route URL displayName: heketi route name name: HEKETI_ROUTE value: heketi-storage - displayName: heketi container image name name: IMAGE_NAME required: true value: rhgs3/rhgs-volmanager-rhel7:v3.10 - description: A unique name to identify this heketi service, useful for running multiple heketi instances displayName: GlusterFS cluster name name: CLUSTER_NAME value: storage
- Execute the following command to delete the deployment configuration, service, and route for heketi:
# oc delete deploymentconfig,service,route heketi-storage
- Execute the following command to get the current HEKETI_ADMIN_KEY.
oc get secret heketi-storage-admin-secret -o jsonpath='{.data.key}'|base64 -d;echo - Execute the following command to deploy the Heketi service, route, and deploymentconfig which will be used to create persistent volumes for OpenShift:
# oc process heketi | oc create -f - service "heketi" created route "heketi" created deploymentconfig "heketi" created
- Execute the following command to verify that the containers are running:
# oc get pods
For example:# oc get pods NAME READY STATUS RESTARTS AGE glusterfs-0h68l 1/1 Running 0 3d glusterfs-0vcf3 1/1 Running 0 3d glusterfs-gr9gh 1/1 Running 0 3d heketi-1-zpw4d 1/1 Running 0 3h storage-project-router-2-db2wl 1/1 Running 0 4d
7.2.4.2. Upgrading Gluster Block if Deployed by Using Ansible
Execute the following steps to upgrade gluster block.
- Execute the following command to upgrade the gluster block:
# yum update gluster-block
- Enable and start the gluster block service:
# systemctl enable gluster-blockd # systemctl start gluster-blockd
- Execute the following command to update the heketi client
# yum update heketi-client -y
- Restart the Heketi service:
# systemctl restart heketi
Note
This step is not applicable if heketi is running as a pod in the Openshift cluster. - If a gluster-block-provisoner-pod already exists then delete it by executing the following commands:
# oc delete dc <gluster-block-dc>
For example:# oc delete dc glusterblock-provisioner-dc
- Edit the glusterblock-provisioner template to change the IMAGE_NAME, IMAGE_VERSION and NAMESPACE.
# oc get templates NAME DESCRIPTION PARAMETERS OBJECTS glusterblock-provisioner glusterblock provisioner template 3 (2 blank) 4 glusterfs GlusterFS DaemonSet template 5 (1 blank) 1 heketi Heketi service deployment template 7 (3 blank) 3
If the template has IMAGE_NAME and IMAGE_VERSION as two separate parameters, then update the glusterblock-provisioner template as following. For example:# oc edit template glusterblock-provisioner - displayName: glusterblock provisioner container image name name: IMAGE_NAME required: true value: rhgs3/rhgs-gluster-block-prov-rhel7 - displayName: glusterblock provisioner container image version name: IMAGE_VERSION required: true value: v3.10 - description: The namespace in which these resources are being created displayName: glusterblock provisioner namespace name: NAMESPACE required: true value: glusterfs - description: A unique name to identify which heketi service manages this cluster, useful for running multiple heketi instances displayName: GlusterFS cluster name name: CLUSTER_NAME value: storageIf the template has only IMAGE_NAME as a parameter, then update the glusterblock-provisioner template as following. For example:# oc edit template glusterblock-provisioner - displayName: glusterblock provisioner container image name name: IMAGE_NAME required: true value: rhgs3/rhgs-gluster-block-prov-rhel7:v3.10 - description: The namespace in which these resources are being created displayName: glusterblock provisioner namespace name: NAMESPACE required: true value: glusterfs - description: A unique name to identify which heketi service manages this cluster, useful for running multiple heketi instances displayName: GlusterFS cluster name name: CLUSTER_NAME value: storage - Delete the following resources from the old pod
# oc delete clusterroles.authorization.openshift.io glusterblock-provisioner-runner
# oc delete serviceaccounts glusterblock-registry-provisioner
- Execute the following command to create a glusterblock-provisioner.
# oc process <gluster_block_provisioner_template> | oc create -f -
7.2.5. Enabling S3 Compatible Object store
Support for S3 compatible Object Store is under technology preview. To enable S3 compatible object store, see https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html-single/operations_guide/#S3_Object_Store.
Note
If you have gluster nodes and heketi pods in glusterfs registry namespace, then follow the steps in section Section 7.3, “Upgrading Gluster Nodes and heketi pods in glusterfs Registry Namespace”.