The following section discusses the contents of setup_ansible.sh and provides an example of an Ansible by Red Hat source system and how to install Ansible by Red Hat on a virtual machine.

Install the following packages on the system performing the provisioning of VMware infrastructure and installation of Red Hat OpenShift.

# ./setup_ansible.sh

# git clone -b vmw-3.6 https://github.com/openshift/openshift-ansible-contrib

# echo "Please fill in your variables ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/ocp-on-vmware.ini"
# echo "Create the initial inventory with the following command ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/ocp-on-vmware.py --create_inventory"
# echo "Create the OCP install vars with the following command ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/ocp-on-vmware.py --create_ocp_vars"
# echo "Lastly, run ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/ocp-on-vmware.py to complete and test the OCP install"

To verify the repository was cloned the tree command can be used to display all of the contents of the git repository.

$ sudo yum -y install tree
$ tree ~/git/

... content abbreviated ...

|-- openshift-ansible-contrib

Before getting started with Ansible, first open ocp-on-vmware.ini and populate the configuration file with your environments information.

$ vim ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/ocp-on-vmware.ini

[vmware]
# unique cluster_id set during script run
cluster_id=
# console port and install type for OpenShift
console_port=8443
deployment_type=openshift-enterprise

# OpenShift Version
openshift_vers=v3_6

# vCenter host address/username and password
vcenter_host=
vcenter_username=administrator@vsphere.local
vcenter_password=

# name of RHEL template to use for OpenShift install
vcenter_template_name=ocp-server-template-2.0.2

# folder/cluster/resource pool in vCenter to organize VMs
vcenter_folder=ocp36
vcenter_datastore=
vcenter_cluster=
vcenter_datacenter=
vcenter_resource_pool=ocp36

# DNS zone where everything will be hosted and app wildcard prefix
public_hosted_zone=
app_dns_prefix=apps

# DNS/gateway/interface name the OpenShift nodes should utilize
vm_dns=
vm_gw=
vm_netmask=
vm_network="VM Network"

# red hat subscription name and password
rhel_subscription_user=
rhel_subscription_pass=

# Internal satellite 6 server
rhel_subscription_server=

# pool with openshift repo access
rhel_subscription_pool=Red Hat OpenShift Container Platform, Premium*

# bringing your own load balancer?
byo_lb=False
lb_host=haproxy-0

# bringing your own NFS server for registry?
byo_nfs=False
nfs_registry_host=nfs-0
nfs_registry_mountpoint=/registry

#create_inventory vars
# number of nodes of each type
master_nodes=3
infra_nodes=3
app_nodes=3
storage_nodes=0

# start node IP address *must be a contiguous space
vm_ipaddr_start=

# node hostname prefix
ocp_hostname_prefix=

# create_ocp_vars vars
# ldap bind user/password and FQDN ldap domain
ldap_user=openshift
ldap_user_password=
ldap_fqdn=

# Deploy OpenShift Metrics
openshift_hosted_metrics_deploy=false

# OpenShift SDN (default value redhat/openshift-ovs-subnet)
openshift_sdn=redhat/openshift-ovs-subnet

# Containerized installation of OpenShift
containerized=false

# persistent container storage: none, crs, cns
container_storage=none

The default values are pre-populated. But, the required values are:

Some explaination of the variables are available Table 3.1, “Red Hat OpenShift Installation Variables” Lastly, your satellite servers address or your Red Hat Network Classic1 username and password for the installation of OpenShift.

DNS is an integral part of a successful Red Hat OpenShift Container Platform deployment/environment.

OpenShift Container Platform requires a properly configured wildcard DNS zone that resolves to the IP address of the OpenShift router. For more information, please refer to the OpenShift Container Platform installation guide. In this reference architecture the --create_inventory parameter on the ocp-on-vmware.py script will help manage DNS records for the OpenShift Container Platform environment.

If applications will be hosted externally, a public zone will be required for the setup. However an internal DNS zone can also be specified instead. This assumes the values have been populated in ocp-on-vmware.ini.

$ ./ocp-on-vmware.py --create_inventory

Configured inventory values:
	master_nodes: 3
	infra_nodes: 3
	app_nodes: 3
	public_hosted_zone: vmware.example.com
	app_dns_prefix: apps
	ocp_hostname_prefix: ocp3-
	byo_nfs: False
	nfs_host: nfs-0
	byo_lb: False
	lb_host: haproxy-0
	vm_ipaddr_start: 10.x.x.224
	Using values from: ./ocp-on-vmware.ini

Continue using these values? [y/N]: y

$ORIGIN apps.vmware.example.com.
*	A	10.x.x.235
$ORIGIN vmware.example.com.
nfs-0	A	10.x.x.224
haproxy-0	A	10.x.x.235
ocp3-master-0	A	10.x.x.225
ocp3-master-1	A	10.x.x.226
ocp3-master-2	A	10.x.x.227
ocp3-app-0	A	10.x.x.228
ocp3-app-1	A	10.x.x.229
ocp3-app-2	A	10.x.x.230
ocp3-infra-0	A	10.x.x.231
ocp3-infra-1	A	10.x.x.232
ocp3-infra-2	A	10.x.x.233

Some important things to note from the above configuration. An HAproxy server will be created for load balancing. That HAproxy server is being assigned the final IP address in the specified IP range as is the wildcard, "*.apps.vmware.example.com." The entries above can be copied and pasted into the DNS server or can be used them as a guideline for record creation.

$ORIGIN apps.vmware.example.com.
*	A	10.x.x.235
$ORIGIN vmware.example.com.
haproxy-0	A	10.x.x.235

There are several options when it comes to authentication of users in Red Hat OpenShift Container Platform. OpenShift can leverage an existing identity provider within an organization such as LDAP, or OpenShift can use external identity providers like GitHub, Google, and GitLab. The configuration of identity providers occurs on the OpenShift master instances. OpenShift allows for multiple identity providers to be specified. This reference architecture document uses LDAP as the authentication provider but any of the other mechanisms would be an acceptable choice. Roles can be added to user accounts to allow for extra privileges, such as the ability to list nodes or assign persistent storage volumes to a project.

For more information on GitHub OAuth and other authentication methods see the Red Hat OpenShift documentation.

The pertinent OpenShift variables for LDAP authentication are listed below:

openshift_master_identity_providers:
- name: Active_Directory
  challenge: true
  login: true
  kind: LDAPPasswordIdentityProvider
  attributes:
    id:
    - dn
    email:
    - mail
    name:
    - cn
    preferredUsername:
    - uid
  insecure: true
  url: "ldap://example.com:389/cn=users,dc=example,dc=com?sAMAccountName"
  bindDN: "cn=openshift,cn=users,dc=example,dc=com"
  bindPassword: "password"

OpenShift needs the fully qualified domain name (FQDN) of the LDAP server in question. An OpenShift user was created in the users’ organizational unit (OU) and assigned the password of password. To use a specific OU for users to authenticate against, create the BIND distinguished name in the desired OU. This provides a way to isolate logins to a specific group by adding an OU and restricting logins to members of that OU.

The ocp-on-vmware.py script that executes the install variables does an LDAP bind and verifies the information provided allows for a successful bind.

The LDAP server configuration can be manually tested (or validated) with the following query:

ldapsearch -x -w password -H ldap://dc1.example.com -b cn=users,dc=example,dc=com -D cn=openshift,cn=users,dc=example,dc=com cn=openshift

NFS is used to store container images in the registry. The NFS server will be created automatically in the playbooks unless it isn’t needed. In that case use the byo_nfs = True option as described below.

There are two methods to provide NFS storage for the registry:

$ vim ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/ocp-on-vmware.ini

... content abbreviated ...

# bringing your own NFS server for registry?
byo_nfs=False
nfs_registry_host=nfs-0
nfs_registry_mountpoint=/registry

... content abbreviated ...

This environment provides an HAproxy service which enables uniform distribution of network traffic across the node instances deployed in VMware. The load balancer will distribute traffic across two different groups

The infra nodes will be utilized to load balance traffic from 80 and 443. The master nodes will be used to load balance our console port 8443. All three of these ports will be transmission control protocol (TCP) connections.

There are clear advantages to doing this:

There are three methods to provide load balancing for the cluster:

$ *vim ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/ocp-on-vmware.ini*

... content abbreviated ...

# bringing your own load balancer?
byo_lb=False
lb_host=haproxy-0

# HA haproxy config
lb_ha_ip=
... content abbreviated ...

Ensure that the proper ports are configured on your existing load balancer and additionally make sure that your wildcard DNS record points to it as well.

An existing port group and virtual LAN (VLAN) are required for deployment. The initial configuration of the Red Hat OpenShift nodes in this reference architecture assumes you will be deploying VMs to a port group called "VM Network". This can be changed in the ocp-on-vmware.ini file under the variable: vm_network. Once deployed vmtoolsd is used to determine the static addresses.

The environment can utilize a virtual dedicated server (vDS) or vSwitch. The specifics of that are unimportant. However, should you wish to utilize network IO control and some of the quality of service (QoS) technologies that VMware employs, you would need to choose a vDS for the deployment.

The vSphere hosts should ultimately have shared storage to house our VMware virtual machine disk files (VMDKs) for our templates. A best practice recommendation would be to enable storage I/O control (SIOC) to address any latency issues caused by performance. The following article discusses in depth how to do this.

This reference architecture assumes some default names as per the ocp-on-vmware.py wrapper script. In particular, vcenter_resource_pool, vcenter_folder and vcenter_template_name all have default values.

The setup playbook in ansible creates a folder and resource pool via the defined values in ocp-on-vmware.ini the defaults are ocp36.

This will allow you to use default names and not force you to customize outside of these entries. If you would like to customize the names feel free to but, remember to specify them later on during the Section 3.1, “Provisioning the Infrastructure with Ansible by Red Hat” section.

A prepared VMware template is used to deploy the environment. In this guide, the Red Hat Enterprise Linux 7 gold image is prepped by starting with an installation of open-vm-tools, perl, open-vm-tools-deploypkg and net-tools.

Any image that meets the prerequisites mentioned above should suffice. The Ansible playbooks will copy over keys or you could prepare the the authorized_keys section ahead of time. The default vcenter_template_name is "ocp3-server-template-2.0.2". Obviously, this can be customized to your environment at runtime.

Prepare the virtual machine to meet the specified requirements listed in the system requirements section of the docs:

A sample script is listed below for preparing the vcenter_template_name:

#!/bin/bash
#
# This file will prepare a RHEL7.4 instance for being converted to a VMware template
# used as an OpenShift Enterprise host

yum install -y open-vm-tools perl open-vm-tools-deploypkg net-tools python-six

systemctl enable vmtoolsd.service
# Remove all let the ansible roles configure
subscription-manager remove --all
subscription-manager unregister

# deploy default ssh key
ssh-keygen -N '' -f ~/.ssh/id_rsa

# This part is redundant provided the ssh-copy-id successfully ran
echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCuY6lBsj95cYTVmzqW2Xci37BJta+ZtNHIee8bFCDskKx/qiH/kbceQGDPpAjREBQBpabhxwd8eUktqdiUI1aQIs7I0Om8XMNUd00Phsz69i8PDzVQyGcLzdn4UPOpS89nFmyto0NJ1V5o4RoR3A7ENbzN7li34g5+zGSBXcVdHFFUErCfOnEtgXw5kltU1byv3+GGKAb+f0CL88BRowp35/NH9sRkP8fzWPS0hl+ofiay5H7xDDd6/nqx6OCd0YHfSEYSTMuRKcM55IFDLLgVNLIumYqwDP6WdCw9dv2aSHPX0bpuLwIEgaMEfGgvTTNi8rZ/rC9Y9OYewfHYOpr9 dphillip@dav1x-m" >> /root/.ssh/authorized_keys

# and run an update
yum update -y && yum clean all

# clean up the system to prepare for being a VMware template
rm -rf /etc/udev/rules.d/70* && rm -rf /etc/ssh/ssh_host_* && logrotate -f /etc/logrotate.conf && rm -rf /var/log/audit/audit.log && history -c
systemctl halt

The master nodes contain the master components, including the API server, controller manager server and ETCD.

The masters:

The masters are used to define routes, services, and volume claims for pods deployed within the OpenShift environment.

The infrastructure nodes are used for the router and registry pods. These nodes could be used if the optional components Kibana and Hawkular metrics are required. The storage for the Docker registry that is deployed on the infrastructure nodes is NFS which allows for multiple pods to use the same storage. NFS storage is used because it largely prevalent in most VMware environments.

The Application nodes are the instances where non-infrastructure based containers run. Depending on the application, VMware specific storage can be applied such as a VMDK which can be assigned using a persistent volume claim for application data that needs to persist between container restarts. A configuration parameter is set on the master which ensures that Red Hat OpenShift Container Platform user containers will be placed on the application nodes by default.

All Red Hat OpenShift Container Platform nodes are assigned a label. This allows certain pods to be deployed on specific nodes. For example, nodes labeled infra are infrastructure nodes. These nodes run the router and registry pods. Nodes with the label app are nodes used for end user application pods.

The configuration parameter 'defaultNodeSelector: "role=app" in /etc/origin/master/master-config.yaml ensures all pods automatically are deployed on application nodes.

As mentioned in the Section 2.8.3, “Authentication” section, authentication for the reference architecture deployment is handled by Microsoft’s Active Directory LDAP (lightweight directory access protocol). The steps below describe how to connect your OpenShift deployment to an LDAP server.

3.1.1.1. Create a Red Hat OpenShift lightweight directory access protocol (LDAP) BIND user account

An existing user account can be utilized or a new account can be created. Below, we have created the user openshift in our default users organizational unit or OU. The location does not matter as our wrapper script will search LDAP for the distinguished name and return our full path to the user account.

Figure 3.1. Active Directory Users and Computers

In our example above, our important OpenShift authentication variables would be:

url: ldap://e2e.bos.redhat.com:389/CN=Users,DC=e2e,DC=bos,DC=redhat,DC=com?sAMAccountName
bindDN: CN=openshift,CN=Users,DC=e2e,DC=bos,DC=redhat,DC=com
bindPassword: password

Let’s assume that the variables will be the following:

$ vim ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/ocp-on-vmware.ini

... content abbreviated ...

# DNS zone where everything will be hosted and app wildcard prefix
public_hosted_zone=example.com
app_dns_prefix=apps

... content abbreviated ...

# create_ocp_vars vars
# ldap bind user/password and FQDN ldap domain
ldap_user=openshift
ldap_user_password=password
ldap_fqdn=example.com

# Deplot OpenShift Metrics
openshift_hosted_metrics_deploy=false

# OpenShift SDN (default value redhat/openshift-ovs-subnet)
openshift_sdn=redhat/openshift-ovs-subnet

... content abbreviated ...

$ cd ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/
./ocp-on-vmware.py --create_ocp_vars

Once the script runs the ocp-install.yaml file that contains the Ansible variables for the OCP installation is modified with above variables

An example is below:

$ vim ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/playbooks/ocp-install.yaml
... ommitted ...
wildcard_zone: apps.example.com
osm_default_subdomain: "{{ wildcard_zone }}"
load_balancer_hostname: haproxy-0.example.com
openshift_master_cluster_hostname: "{{ load_balancer_hostname }}"
openshift_master_cluster_public_hostname: "{{ load_balancer_hostname }}"
openshift_master_identity_providers:
- name: Active_Directory
  challenge: true
  login: true
  kind: LDAPPasswordIdentityProvider
  attributes:
    id:
    - dn
    email:
    - mail
    name:
    - cn
    preferredUsername:
    - uid
  insecure: true
  url: ldap://example.com:389/CN=Users,DC=example,DC=com?sAMAccountName
  bindDN: CN=openshift,CN=Users,DC=example,DC=com
  bindPassword: password
... ommitted ...

Let’s assume that our variables will be the following:

$ vim ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/ocp-on-vmware.ini

... content abbreviated ...
public_hosted_zone=example.com

#create_inventory vars
# number of nodes of each type
master_nodes=3
infra_nodes=3
app_nodes=3

# start node IP address must be a contiguous space
vm_ipaddr_start=10.x.x.224

# node hostname prefix
ocp_hostname_prefix=ocp3-

... content abbreviated ...

$ cd ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/
./ocp-on-vmware.py --create_inventory --no-confirm

# Here is what should go into your DNS records
$ORIGIN apps.example.com.
*                   A       10.x.x.234
$ORIGIN example.com.
nfs-0	             	A       10.x.x.224
haproxy-0           A       10.x.x.234
ocp3-master-0     	A       10.x.x.225
ocp3-master-1     	A       10.x.x.226
ocp3-master-2     	A       10.x.x.227
ocp3-app-0	       	A       10.x.x.228
ocp3-app-1	       	A       10.x.x.229
ocp3-app-2		      A       10.x.x.230
ocp3-infra-0        A       10.x.x.231
ocp3-infra-1        A       10.x.x.232
ocp3-infra-2        A       10.x.x.233
# Please note, if you have chosen to bring your own load balancer and NFS Server you will need to ensure that
# these records are added to DNS and properly resolve.

As you can see based on our input we have the guidelines for our DNS zone creation or modification. Also, the wildcard_zone record is shown with our supplied public_hosted_zone and the default app_dns_prefix of apps. Additionally, in that directory we have created a dynamic base inventory file infrastructure.json with the specifics to provisioning the environment.

Here is an interesting excerpt from infrastructure.json converted to YAML:

ocp3-master-0:
  guestname: ocp3-master-0
  tag: 3e06olsus6x2rbgcxw4t-master
  ip4addr: 10.19.114.225

Note the "tag: 3e06olsus6x2rbgcxw4t-master" in the entry. This will be the annotation created on the virtual machine and is how OpenShift labels are generated for the VMware virtual machines.

Intelligence is built into the playbooks to allow for certain variables to be set using options provided by the ocp-on-vmware.py script. The script allows for deployment into an existing environment (brownfield) or a new environment (greenfield) using a series of Ansible playbooks. Once the Ansible playbooks begin, the installation automatically flows from the VMware deployment to the Red Hat OpenShift deployment and post installation tasks.

$ cd ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/
$ vim ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/ocp-on-vmware.ini

[vmware]
# console port and install type for OpenShift
console_port=8443
deployment_type=openshift-enterprise

# vCenter host address/username and password
vcenter_host=
vcenter_username=administrator@vsphere.local
vcenter_password=

... ommitted ...

$ cd ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/
$ ./ocp-on-vmware.py

Configured values:

console_port: 8443
deployment_type: openshift-enterprise
vcenter_host: 10.x.x.25
vcenter_username: administrator@vsphere.local
vcenter_password:
vcenter_template_name: ocp-server-template-2.0.2
vcenter_folder: ocp36
vcenter_cluster: devel
vcenter_datacenter: Boston
vcenter_datastore: ose3-vmware-prod
vcenter_resource_pool: ocp36
public_hosted_zone: vcenter.example.com
app_dns_prefix: apps
vm_dns: 10.x.x.5
vm_gw: 10.x.x.254
vm_netmask: 255.255.255.0
vm_network: VM Network
byo_lb: False
lb_host: haproxy-0.vcenter.example.com
byo_nfs: False
nfs_registry_host: nfs-0.vcenter.example.com
nfs_registry_mountpoint: /registry
apps_dns: apps.vcenter.example.com
openshift_sdn: redhat/openshift-ovs-subnet
openshift_hosted_metrics_deploy: false
container_storage: none

Using values from: ./ocp-on-vmware.ini

Continue using these values? [y/N]:

$ cd ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/
$ ./ocp-on-vmware.py --tag nfs,haproxy,ocp-install,ocp-configure

Configured values:

cluster_id: my_custom_id
console_port: 8443
deployment_type: openshift-enterprise
vcenter_host: 10.x.x.25
vcenter_username: administrator@vsphere.local
vcenter_password:
vcenter_template_name: ocp-server-template-2.0.2
vcenter_folder: ocp36
vcenter_cluster: devel
vcenter_datacenter: Boston
vcenter_datastore: ose3-vmware-prod
vcenter_resource_pool: ocp36
public_hosted_zone: vcenter.example.com
app_dns_prefix: apps
vm_dns: 10.x.x.5
vm_gw: 10.x.x.254
vm_netmask: 255.255.255.0
vm_network: VM Network
rhel_subscription_user: rhn_user
rhel_subscription_password:
byo_lb: False
lb_host: haproxy-0.vcenter.example.com
byo_nfs: False
nfs_registry_host: nfs-0.vcenter.example.com
nfs_registry_mountpoint: /registry
apps_dns: apps.vcenter.example.com
Using values from: ./ocp-on-vmware.ini

Continue using these values? [y/N]:

$ cd ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/
$ vim ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/ocp-on-vmware.ini

... content abbreviated ...

# bringing your own load balancer?
byo_lb=True
lb_host=my-load-balancer.lb.example.com

# bringing your own NFS server for registry?
byo_nfs=True
nfs_registry_host=my-nfs-server.nfs.example.com
nfs_registry_mountpoint=/my-registry

... content abbreviated ...

$ ./ocp-on-vmware.py --tag ocp-install,ocp-configure

Configured values:

cluster_id: my_custom_id
console_port: 8443
deployment_type: openshift-enterprise
vcenter_host: 10.19.114.25
vcenter_username: administrator@vsphere.local
vcenter_password:
vcenter_template_name: ocp-server-template-2.0.2
vcenter_folder: ocp
vcenter_cluster: devel
vcenter_datacenter: Boston
vcenter_resource_pool: OCP3
public_hosted_zone: vcenter.e2e.bos.redhat.com
app_dns_prefix: apps
vm_dns: 10.19.114.5
vm_gw: 10.19.115.254
vm_netmask: 255.255.254.0
vm_network: VM Network
byo_lb: True
lb_host: my-load-balancer
byo_nfs: True
nfs_host: my-nfs-server
nfs_registry_mountpoint: /my-registry
apps_dns: apps.vcenter.e2e.bos.redhat.com
Using values from: ./ocp-on-vmware.ini

Continue using these values? [y/N]:

3.3.3.4. Post Ansible Deployment

Prior to Chapter 4, Operational Management, create DRS anti-affinity rules to ensure maximum availability for our cluster.

1. Open the VMware vCenter web client, select the cluster, choose configure.

1. Under Configuration, select VM/Host Rules.

1. Click add, and create a rules to keep the masters separate.

The following VMware documentation goes over creating and configuring anti-affinity rules in depth.

Once the playbooks have successfully completed, the next steps will be to perform the steps defined in Chapter 4, Operational Management. In the event that OpenShift failed to install, follow the steps in Appendix C: Appendix E, Installation Failure to restart the installation of OpenShift.

Lastly, set all of the VMs created to High VM Latency to ensure some additional tuning recommended by VMware for latency sensitive workloads as described here.

1. Open the VMware vCenter web client and under the virtual machines summary tab, in the 'VM Hardware' box select 'Edit Settings'.
2. Under, 'VM Options', expand 'Advanced'.
3. Select the 'Latency Sensitivity' dropbox and select 'High'.

Figure 3.2. VMware High Latency

3.3.3.4.1. Registry Console Selector (Optional)

The OpenShift Registry Console deployment is deployed on any Red Hat OpenShift Container Platform node by default, so the container may end up running on any of the application nodes.

From the first master instance(ocp3-master-0.example.com), ensure the OpenShift Registry Console pod runs on the infra nodes by modifying the nodeSelector as follows:

$ oc patch dc registry-console \
    -n default \
    -p '{"spec":{"template":{"spec":{"nodeSelector":{"role":"infra"}}}}}'

Note

There is a bugzilla ID: 1425022 being investigated by Red Hat at the time of writing this paper to fix this issue.

If you try to run the following command, it should fail.

# oc get nodes --show-labels
Error from server: User "user@redhat.com" cannot list all nodes in the cluster

The reason it is failing is because the permissions for that user are incorrect. Get the username and configure the permissions.

$ oc whoami

Once the username has been established, log back into a master node and enable the appropriate permissions for your user. Perform the following step from the first master (ocp3-master-0.example.com).

# oc adm policy add-cluster-role-to-user cluster-admin CN=openshift,CN=Users,DC=e2e,DC=bos,DC=redhat,DC=com

Attempt to list the nodes again and show the labels.

# oc get nodes --show-labels
NAME                        STATUS                     AGE       VERSION             LABELS
ocp3-app-0.example.com      Ready                      2d        v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ocp3-app-0.example.com,role=app
ocp3-app-1.example.com      Ready                      2d        v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ocp3-app-1.example.com,role=app
ocp3-app-2.example.com      Ready                      2d        v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ocp3-app-2.example.com,role=app
ocp3-infra-0.example.com    Ready                      2d        v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ocp3-infra-0.example.com,role=infra
ocp3-infra-1.example.com    Ready                      2d        v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ocp3-infra-1.example.com,role=infra
ocp3-infra-2.example.com    Ready                      2d        v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ocp3-infra-2.example.com,role=infra
ocp3-master-0.example.com   Ready,SchedulingDisabled   2d        v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ocp3-master-0.example.com,role=master
ocp3-master-1.example.com   Ready,SchedulingDisabled   2d        v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ocp3-master-1.example.com,role=master
ocp3-master-2.example.com   Ready,SchedulingDisabled   2d        v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ocp3-master-2.example.com,role=master

List the router and registry by changing to the default project.

# oc project default
# oc get all
# oc status
In project default on server https://haproxy-0.example.com:8443

https://docker-registry-default.apps.vcenter.e2e.bos.redhat.com (passthrough) (svc/docker-registry)
  dc/docker-registry deploys docker.io/openshift3/ose-docker-registry:v3.6.173.0.21
    deployment #1 deployed 23 hours ago - 1 pod

svc/kubernetes - 172.30.0.1 ports 443->8443, 53->8053, 53->8053

https://registry-console-default.apps.vcenter.e2e.bos.redhat.com (passthrough) (svc/registry-console)
  dc/registry-console deploys registry.access.redhat.com/openshift3/registry-console:v3.6
    deployment #1 deployed 23 hours ago - 1 pod

svc/router - 172.30.253.65 ports 80, 443, 1936
  dc/router deploys docker.io/openshift3/ose-haproxy-router:v3.6.173.0.21
    deployment #1 deployed 23 hours ago - 1 pod

View details with 'oc describe <resource>/<name>' or list everything with 'oc get all'.

Observe the output of oc get all and oc status. Notice that the registry and router information is clearly listed.

The OpenShift Ansible playbooks configure two infrastructure nodes that have two registries running. In order to understand the configuration and mapping process of the registry pods, the command 'oc describe' is used. oc describe details how registries are configured and mapped to the NFS mount for storage. Using oc describe should help explain how HA works in this environment.

$ oc describe svc/docker-registry
Name:			docker-registry
Namespace:		default
Labels:			docker-registry=default
Selector:		docker-registry=default
Type:			ClusterIP
IP:			172.30.252.119
Port:			5000-tcp	5000/TCP
Endpoints:		172.16.5.3:5000,172.16.8.2:5000
Session Affinity:	ClientIP
No events.

Notice that the registry has two endpoints listed. Each of those endpoints represents a Docker container. The ClusterIP listed is the actual ingress point for the registries.

Once the endpoints are known, go to one of the infra nodes running a registry and grab some information about it. Capture the container UID in the leftmost column of the output.

# oc get pods
NAME                      READY     STATUS    RESTARTS   AGE
docker-registry-2-8b7c6   1/1       Running   0          2h
docker-registry-2-drhgz   1/1       Running   0          2h                          k8s_registry.90479e7d_docker-registry-2-jueep_default_d5882b1f-5595-11e6-a247-0eaf3ad438f1_ffc47696

# oc exec docker-registry-2-8b7c6 cat /config.yml
  version: 0.1
  log:
    level: debug
  http:
    addr: :5000
  storage:
    cache:
      blobdescriptor: inmemory
    filesystem:
      rootdirectory: /registry
    delete:
      enabled: true
  auth:
    openshift:
      realm: openshift

      # tokenrealm is a base URL to use for the token-granting registry endpoint.
      # If unspecified, the scheme and host for the token redirect are determined from the incoming request.
      # If specified, a scheme and host must be chosen that all registry clients can resolve and access:
      #
      # tokenrealm: https://example.com:5000
  middleware:
    registry:
      - name: openshift
    repository:
      - name: openshift
        options:
          acceptschema2: false
          pullthrough: true
          enforcequota: false
          projectcachettl: 1m
          blobrepositorycachettl: 10m
    storage:
      - name: openshift

Additionally, registry information can be garnered via the oc command as well.

oc volume dc/docker-registry
deploymentconfigs/docker-registry
  pvc/registry-claim (allocated 20GiB) as registry-storage
    mounted at /registry
  secret/registry-certificates as registry-certificates
    mounted at /etc/secrets

This section will explore the Docker storage on an infrastructure node.

The example below can be performed on any node, but for this example the infrastructure node(ocp-infra-0.example.com) is used.

The output below verifies Docker storage is not using a loop back device.

# docker info
...omitted...

 Server Version: 1.12.6
 Storage Driver: overlay2
  Backing Filesystem: xfs
 Logging Driver: journald
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host null overlay
  Authorization: rhel-push-plugin
 Swarm: inactive
 Runtimes: runc docker-runc
 Default Runtime: docker-runc
 Security Options: seccomp selinux
 Kernel Version: 3.10.0-693.el7.x86_64
 Operating System: OpenShift Enterprise
 OSType: linux
 Architecture: x86_64
 Number of Docker Hooks: 3
 CPUs: 2
 Total Memory: 15.5 GiB
 Name: master-0
 ID: DTNQ:U22B:R6GN:2ZES:AATU:SYVE:SYQU:6FKS:GATT:I4TI:CV3H:6MJ2
 Docker Root Dir: /var/lib/docker
...omitted...
Registries: registry.access.redhat.com (secure), docker.io (secure)

Verify three disks are attached to the instance. The disk /dev/sda is used for the OS, /dev/sdb is used for Docker storage, and /dev/sdc is used for EmptyDir storage for containers that do not use a persistent volume.

# fdisk -l

Disk /dev/sdb: 42.9 GB, 42949672960 bytes, 83886080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048    83886079    41942016   8e  Linux LVM

Disk /dev/sdc: 42.9 GB, 42949672960 bytes, 83886080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sda: 42.9 GB, 42949672960 bytes, 83886080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos

Log into the VMware vCenter client. Under VMs and Templates, locate your running ocp3-master-2.example.com VM, select it, right click and change the state to Power off.

Ensure the console can still be accessed by opening a browser and accessing haproxy-0.example.com. At this point, the cluster is in a degraded state because only two out of three master nodes are running.

SSH into the first master node (ocp3-master-0.example.com). Using the output of the command hostname, issue the etcdctl command to confirm that the cluster is healthy.

$ ssh root@ocp3-master-0.example.com

# hostname
ip-10-20-1-106.ec2.internal
# etcdctl -C https://$(hostname):2379 --ca-file=/etc/origin/master/master.etcd-ca.crt --cert-file=/etc/origin/master/master.etcd-client.crt --key-file=/etc/origin/master/master.etcd-client.key cluster-health
member d3525253178d331c is healthy: got healthy result from https://10.19.114.225:2379
failed to check the health of member edf71ee725ea87b6 on https://10.19.114.226:2379: Get https://10.19.114.226:2379/health: dial tcp 10.19.114.226:2379: i/o timeout
member edf71ee725ea87b6 is unreachable: [https://10.19.114.226:2379] are all unreachable
member f2e1170c11b5cea8 is healthy: got healthy result from https://10.19.114.227:2379

Notice how one member of the etcd cluster is now unreachable.

Restart ocp3-master-2.example.com by following the same steps in the VMWare console as noted above.

This section shows what to expect when an infrastructure node fails or is brought down intentionally.

$ oc get projects
NAME               DISPLAY NAME   STATUS
openshift                         Active
openshift-infra                   Active
ttester                           Active
test-app1                         Active
default                           Active
management-infra                  Active

$ oc project test-app1
Now using project "test-app1" on server "https://haproxy-0.example.com".

$ oc status
In project test-app1 on server https://haproxy-0.example.com

http://php-test-app1.apps.sysdeseng.com to pod port 8080-tcp (svc/php-prod)
  dc/php-prod deploys istag/php-prod:latest <-
    bc/php-prod builds https://github.com/openshift/cakephp-ex.git with openshift/php:5.6
    deployment #1 deployed 27 minutes ago - 1 pod

# oc whoami -t
feAeAgL139uFFF_72bcJlboTv7gi_bo373kf1byaAT8

# docker pull fedora/apache
# docker images

# oc get svc
NAME               CLUSTER-IP       EXTERNAL-IP   PORT(S)                   AGE
docker-registry    172.30.195.135   <none>        5000/TCP                  23h
kubernetes         172.30.0.1       <none>        443/TCP,53/UDP,53/TCP     23h
registry-console   172.30.203.227   <none>        9000/TCP                  23h
router             172.30.253.65    <none>        80/TCP,443/TCP,1936/TCP   23h

# docker tag docker.io/fedora/apache 172.30.252.119:5000/openshift/prodapache

# docker images

# docker login -u CN=openshift,CN=Users,DC=e2e,DC=bos,DC=redhat,DC=com 172.30.195.135:5000

# oc adm policy add-role-to-user admin CN=openshift,CN=Users,DC=e2e,DC=bos,DC=redhat,DC=com -n openshift
# oc adm policy add-role-to-user system:registry CN=openshift,CN=Users,DC=e2e,DC=bos,DC=redhat,DC=com
# oc adm policy add-role-to-user system:image-builder CN=openshift,CN=Users,DC=e2e,DC=bos,DC=redhat,DC=com

# docker push 172.30.195.135:5000/openshift/prodapache
The push refers to a repository [172.30.195.135:5000/openshift/prodapache]
389eb3601e55: Layer already exists
c56d9d429ea9: Layer already exists
2a6c028a91ff: Layer already exists
11284f349477: Layer already exists
6c992a0e818a: Layer already exists
latest: digest: sha256:ca66f8321243cce9c5dbab48dc79b7c31cf0e1d7e94984de61d37dfdac4e381f size: 6186

# oc project default
Now using project "default" on server "https://haproxy-0.example.com".

# oc get pods
NAME                       READY     STATUS    RESTARTS   AGE
docker-registry-1-065t9    1/1       Running   0          23h
registry-console-1-h7vsc   1/1       Running   0          23h
router-1-ksxk1             1/1       Running   0          23h

# oc describe pod docker-registry-1-065t9 | grep -i node
Node:			infra-0/10.19.114.228
Node-Selectors:	role=infra
# oc describe pod router-1-ksxk1 | grep -i node
Node:			infra-0/10.19.114.228
Node-Selectors:	role=infra

# oc describe pod docker-registry-1-065t9 | grep -i node
Node:			infra-1/10.19.114.229
Node-Selectors:	role=infra
# oc describe pod router-1-ksxk1 | grep -i node
Node:			infra-1/10.19.114.229
Node-Selectors:	role=infra

From the workstation that was used to clone the openshift-ansible-contrib repo run the following to ensure that the newest openshift-ansible playbooks and roles are available and to perform the minor upgrade against the deployed environment.

# yum update atomic-openshift-utils ansible
$ cd ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/
$ ocp-on-vmware.py --tag ocp-update*

The openshift-minor-update.yaml playbook will not restart the instances after updating occurs. Restarting the nodes including the masters can be completed by adding the following line to the minor-update.yaml playbook.

$ cd ~/git/openshift-ansible-contrib/playbooks
$ vi minor-update.yaml
    openshift_rolling_restart_mode: system

The deployed OpenShift environment may not be the latest major version of OpenShift. The minor-update.yaml allows for a variable to be passed to perform an upgrade on previous versions. Below is an example of performing the upgrade on a 3.6 environment.

# yum update atomic-openshift-utils ansible
$ cd ~/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/
$ vi ocp-on-vmware.ini
# OpenShift Version
openshift_vers=v3_6
$ ocp-on-vmware.py --create_ocp_vars
$ ocp-on-vmware.py --tag ocp-update

As stated in the previous section a StorageClass is created and configured when using the ocp-on-vmware script. To view the configuration of the StorageClass log into the first master node and view the file /root/cloud-provider-storage-class.yaml.

$ cat cloud-provider-storage-class.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: "ose3-vmware-prod"
provisioner: kubernetes.io/vsphere-volume
parameters:
    diskformat: zeroedthick
    datastore: "ose3-vmware-prod"

Since the StorageClass object is created at default. The oc command can be used
to verify the StorageClass exists.

$ oc get sc
NAME               TYPE
ose3-vmware-prod   kubernetes.io/vsphere-volume

$ oc describe sc ose3-vmware-prod
Name:		ose3-vmware-prod
IsDefaultClass:	No
Annotations:	<none>
Provisioner:	kubernetes.io/vsphere-volume
Parameters:	datastore=ose3-vmware-prod,diskformat=zeroedthick
Events:		<none>

With the StorageClass object created OpenShift can now dynamically provision VMDKs for persistent storage for containers within the OpenShift environment.

$ vi storage-class-vmware-claim.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: ose3-vmware-prod
  annotations:
    volume.beta.kubernetes.io/storage-class: ose3-vmware-prod
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

$ oc create -f storage-class-vmware-claim.yaml
$ oc describe pvc ose3-vmware-prod

Name:		ose3-vmware-prod
Namespace:	default
StorageClass:	ose3-vmware-prod
Status:		Bound
Volume:		pvc-cc8a9970-7c76-11e7-ae86-005056a571ee
Labels:		<none>
Annotations:	pv.kubernetes.io/bind-completed=yes
		pv.kubernetes.io/bound-by-controller=yes
		volume.beta.kubernetes.io/storage-class=vmware-datastore-ssd
		volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/vsphere-volume
Capacity:	2Gi
Access Modes:	RWO
Events:
  FirstSeen	LastSeen	Count	From				SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----				-------------	--------	------			-------
  19s		19s		1	persistentvolume-controller			Normal		ProvisioningSucceeded	Successfully provisioned volume pvc-cc8a9970-7c76-11e7-ae86-005056a571ee using kubernetes.io/vsphere-volume

Now, in vCenter, a couple of changes are initiated:

Here the new disk is created.

Secondly, the disk is ready to be consumed by a VM to be attached to a POD.

While datastores are generally accessible via shared storage to all the nodes of your cluster, the VMDKs are tied to a specific machine. This explains the ReadWriteOnce limitation of the persistent storage.

The persistent volume claim (PVC) will change the pod from using EmptyDir non-persistent storage to storage backed by an NFS volume or PV as created above. The PVC will be created as long as the storage size is equal or greater to the PV and the accessModes are the same (i.e., ReadWriteOnce).

$ vi nfs-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: db
spec:
  accessModes:
  - ReadWriteOnce
  resources:
     requests:
       storage: 5Gi


$ oc create -f nfs-pvc.yaml
persistentvolumeclaim "db" created

There may become a point in which a PVC is no longer necessary for a project. The following can be done to remove the PVC.

$ oc delete pvc db
persistentvolumeclaim "db" deleted
$ oc get pvc db
No resources found.
Error from server: persistentvolumeclaims "db" not found

Deployment of Container-Native Storage (CNS) on OpenShift Container Platform (OCP) requires at least three OpenShift nodes with at least one unused block storage device attached on each of the nodes. Dedicating three OpenShift nodes to CNS will allow for the configuration of one StorageClass object to be used for applications. If two types of StorageClass objects are required (e.g. HDD and SSD types) then a minimum of six CNS nodes must be deployed and configured. This is because only a single CNS container per OpenShift node is supported.

If the CNS instances will serve dual roles such as hosting application pods and glusterfs pods ensure the instances have enough resources to support both operations. CNS hardware requirements state that there must be 32GB of RAM per node or virtual machine. There is a current limit of 300 volumes or PVs per 3 node CNS cluster.

A python script named add-node.py is provided in the openshift-ansible-contrib git repository which will deploy three nodes or virtual machines, add the virtual machines to the OpenShift environment with specific OCP labels and add a VMDK volume to each node as an available block device to be used for CNS.

Do the following from the workstation performing the deployment of the OCP Reference Architecture. The ocp-on-vmware.ini file used to create the OpenShift deployment must be in the directory where add-node.py is ran from. There are two entries in the ocp-on-vmware.ini file that must be added or modified. They are the vcenter_datastore and container_storage. The VMware datastore entered is where all of the new OCP CNS nodes or virtual machines will be stored so verify that this datastore has at least 1TB of available storage.

$ cd /root/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/
$ cat ocp-on-vmware.ini
….omitted….
# folder/cluster/resource pool in vCenter to organize VMs
vcenter_folder=ocp3
vcenter_datastore=DPLHP380G9-10-SS200-2
vcenter_cluster=OCP3
vcenter_resource_pool=OCP3
vcenter_datacenter=vDPL
….omitted….
# persistent container storage: none, crs, cns
container_storage=cns

$ ./add-node.py --node_type=storage
Configured inventory values:
	 console_port:  8443
	 deployment_type:  openshift-enterprise
	 openshift_vers:  v3_6
	 vcenter_host:  172.0.10.246
	 vcenter_username:  administrator@vsphere.local
	 vcenter_password: *******
	 vcenter_template_name:  ocp-server-template-2.0.2-new
	 vcenter_folder:  ocp3
	 vcenter_datastore:  DPLHP380G9-10-SS200-2
	 vcenter_cluster:  OCP3
	 vcenter_resource_pool:  OCP3
	 vcenter_datacenter:  vDPL
	 public_hosted_zone:  dpl.local
	 app_dns_prefix:  apps
	 vm_dns:  172.0.10.241
	 vm_gw:  172.0.10.2
	 vm_netmask:  255.255.255.0
	 vm_network:  "Private"
	 rhel_subscription_user: rhn_user
	 rhel_subscription_pass: *******
	 rhel_subscription_server:
	 rhel_subscription_pool:  Red Hat OpenShift Container Platform, Premium*
	 byo_lb:  False
	 lb_host:  haproxy-0
	 byo_nfs:  False
	 nfs_host:  nfs-0
	 nfs_registry_mountpoint:  /exports
	 master_nodes:  3
	 infra_nodes:  3
	 app_nodes:  6
	 storage_nodes:  0
	 vm_ipaddr_start:  172.0.10.201
	 ocp_hostname_prefix:  ocp3-
	 auth_type:  ldap
	 ldap_user:  openshift
	 ldap_user_password: *******
	 ldap_fqdn:  dpl.local
	 openshift_hosted_metrics_deploy:  false
	 openshift_sdn:  redhat/openshift-ovs-subnet
	 containerized:  false
	 container_storage:  cns
	 tag:  None
	 node_number:  1
	 ini_path:  ./ocp-on-vmware.ini
	 node_type:  storage

Continue creating the inventory file with these values? [y/N]: y
Gluster topology file created using /dev/sdd: topology.json
Inventory file created: add-node.json
host_inventory:
  ocp3-app-cns-0:
    guestname: ocp3-app-cns-0
    ip4addr: 172.0.10.211
    tag: storage
  ocp3-app-cns-1:
    guestname: ocp3-app-cns-1
    ip4addr: 172.0.10.212
    tag: storage
  ocp3-app-cns-2:
    guestname: ocp3-app-cns-2
    ip4addr: 172.0.10.213
    tag: storage

Continue adding nodes with these values? [y/N]:

The correct firewall ports are automatically applied on the nodes deployed using the add-node.py with option --node_type=storage script. If the script has not been used to create the new OCP CNS nodes then the ports will need to be configured manually. On each of the OCP nodes that will host the Red Hat Gluster Storage container, add the following rules to /etc/sysconfig/iptables and reload the iptables:

$ cat /etc/sysconfig/iptables
….omitted….
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 24007 -j ACCEPT
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 24008 -j ACCEPT
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 2222 -j ACCEPT
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m multiport --dports 49152:49664 -j ACCEPT
….omitted….

# systemctl reload iptables

These activities should be done on the master due to the requirement of setting the node selector. The account performing the CNS activities must be a cluster-admin. Example shown below for the openshift user.

$ oc adm policy add-cluster-role-to-user cluster-admin cn=openshift,cn=users,dc=example,dc=com

The project name used for this example will be storage but the project name can be whatever value an administrator chooses.

If the CNS nodes will only be used for CNS then a node-selector should be supplied.

$ oc adm new-project storage --node-selector='role=storage'

If the CNS nodes will serve the role of being used for both CNS and application pods then a node-selector does not need to supplied.

$ oc adm new-project storage

An oc adm policy must be set to enable the deployment of the privileged containers as Red Hat Gluster Storage containers can only run in the privileged mode.

$ oc project storage
$ oc adm policy add-scc-to-user privileged -z default

Perform the following steps from CLI on a local or deployment workstation and ensure that the oc client has been installed and configured. An entitlement for Red Hat Gluster Storage is required to install the Gluster services.

# subscription-manager repos --enable=rh-gluster-3-for-rhel-7-server-rpms
# subscription-manager repos --enable=rhel-7-server-rpms
# yum install -y cns-deploy heketi-client

The Container-Native Storage glusterfs and heketi pods, services, and heketi route are created using the cns-deploy tool which was installed during the prerequisite step.

A heketi topology file is used to create the Trusted Storage Pool. The topology describes the OpenShift nodes that will host Red Hat Gluster Storage services and their attached storage devices. A sample topology file topology-sample.json is installed with the heketi-client package in the /usr/share/heketi/ directory.

Below is an example of 3 node topology.json file with /dev/sdd as the VMware volume or device used for CNS. This file, topology.json is created and placed in the directory where add-node.py --node_type=storage is issued from.

Edit the values of node.hostnames.manage, node.hostnames.storage, and devices in the topology.json file based on the the OCP nodes that have been deployed in the previous step if the add-node.py script was not used.

$ vi topology.json
{
    "clusters": [
        {
            "nodes": [
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ocp3-app-cns-0.dpl.local"
                            ],
                            "storage": [
                                "172.0.10.211"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/sdd"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ocp3-app-cns-1.dpl.local"
                            ],
                            "storage": [
                                "172.0.10.212"
                            ]
                        },
                        "zone": 2
                    },
                    "devices": [
                        "/dev/sdd"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ocp3-app-cns-2.dpl.local"
                            ],
                            "storage": [
                                "172.0.10.213"
                            ]
                        },
                        "zone": 3
                    },
                    "devices": [
                        "/dev/sdd"
                    ]
                }
            ]
        }
    ]
}

Ensure that the storage project is the current project.

$ oc project storage
Already on project "storage" on server "https://ocp3-haproxy-0.dpl.local:8443".

To launch the deployment of CNS the script cns-deploy will be used. It is advised to specify an admin-key and user-key for security reasons when launching the topology. Both admin-key and user-key are user defined values, they do not exist before this step. The heketi admin key (password) will later be used to create a heketi-secret in OCP. Be sure to note these values as they will be needed in future operations. The cns-deploy script will prompt the user before proceeding.

$ cns-deploy -n storage -g topology.json --admin-key 'myS3cr3tpassw0rd' --user-key 'mys3rs3cr3tpassw0rd'
Welcome to the deployment tool for GlusterFS on Kubernetes and OpenShift.

Before getting started, this script has some requirements of the execution
environment and of the container platform that you should verify.

The client machine that will run this script must have:
 * Administrative access to an existing Kubernetes or OpenShift cluster
 * Access to a python interpreter 'python'
 * Access to the heketi client 'heketi-cli'

Each of the nodes that will host GlusterFS must also have appropriate firewall
rules for the required GlusterFS ports:
 * 2222  - sshd (if running GlusterFS in a pod)
 * 24007 - GlusterFS Daemon
 * 24008 - GlusterFS Management
 * 49152 to 49251 - Each brick for every volume on the host requires its own
   port. For every new brick, one new port will be used starting at 49152. We
   recommend a default range of 49152-49251 on each host, though you can adjust
   this to fit your needs.

In addition, for an OpenShift deployment you must:
 * Have 'cluster_admin' role on the administrative account doing the deployment
 * Add the 'default' and 'router' Service Accounts to the 'privileged' SCC
 * Have a router deployed that is configured to allow apps to access services
   running in the cluster

Do you wish to proceed with deployment?

[Y]es, [N]o? [Default: Y]: Y
Using OpenShift CLI.
NAME           STATUS    AGE
storage   Active    32m
Using namespace "storage".
Checking that heketi pod is not running ... OK
template "deploy-heketi" created
serviceaccount "heketi-service-account" created
template "heketi" created
template "glusterfs" created
role "edit" added: "system:serviceaccount:storage:heketi-service-account"
node "ocp3-app-cns-0.dpl.local" labeled
node "ocp3-app-cns-1.dpl.local" labeled
node "ocp3-app-cns-2.dpl.local" labeled
daemonset "glusterfs" created
Waiting for GlusterFS pods to start ... OK
service "deploy-heketi" created
route "deploy-heketi" created
deploymentconfig "deploy-heketi" created
Waiting for deploy-heketi pod to start ... ^[[AOK
Creating cluster ... ID: 8578ad5529c354e9d21898cba4c4a1c9
Creating node ocp3-app-cns-0.dpl.local ... ID: 8424caa3b7bd9e9098ef200fe8033edf
Adding device /dev/sdd ... OK
Creating node ocp3-app-cns-1.dpl.local ... ID: ca9344f2390798304b1a7877ecc0bb85
Adding device /dev/sdd ... OK
Creating node ocp3-app-cns-2.dpl.local ... ID: ca2a1703a3ed68979eef307abe2f1770
Adding device /dev/sdd ... OK
heketi topology loaded.
Saving heketi-storage.json
secret "heketi-storage-secret" created
endpoints "heketi-storage-endpoints" created
service "heketi-storage-endpoints" created
job "heketi-storage-copy-job" created
deploymentconfig "deploy-heketi" deleted
route "deploy-heketi" deleted
service "deploy-heketi" deleted
job "heketi-storage-copy-job" deleted
pod "deploy-heketi-1-cjt16" deleted
secret "heketi-storage-secret" deleted
service "heketi" created
route "heketi" created
deploymentconfig "heketi" created
Waiting for heketi pod to start ... OK
heketi is now running.
Ready to create and provide GlusterFS volumes.

After successful deploy validate that there are now 3 glusterfs pods and 1 heketi pod in the storage project.

$ oc get pods -o=wide
NAME                     READY     STATUS    RESTARTS   AGE     IP              NODE
glusterfs-496n0          1/1       Running   0          14m     172.16.2.4      ocp3-app-cns-0
glusterfs-hx6z1          1/1       Running   0          14m     172.16.3.4      ocp3-app-cns-1
glusterfs-mtr2t          1/1       Running   0          14m     172.16.4.4      ocp3-app-cns-2
heketi-1-xllk8           1/1       Running   0          9m      172.16.6.2      ocp3-app-cns-0

A new route will be created for the heketi service that was deployed during the run of the cns-deploy script. The heketi route URL is used by the heketi-client. The same route URL will be used to create StorageClass objects.

The first step is to find the endpoint for the heketi service and then set the environment variables for the route of the heketi server, the heketi cli user, and the heketi cli key.

$ oc get routes heketi
NAME      HOST/PORT     PATH      SERVICES   PORT      TERMINATION   WILDCARD
heketi    heketi-storage.apps.dpl.local             heketi     <all>                   None

$ export HEKETI_CLI_SERVER=http://heketi-storage.apps.dpl.local
$ export HEKETI_CLI_USER=admin
$ export HEKETI_CLI_KEY=myS3cr3tpassw0rd

To validate that heketi loaded the topology and has the cluster created execute the following commands:

$ heketi-cli topology info
... ommitted ...
$ heketi-cli cluster list
Clusters:
8578ad5529c354e9d21898cba4c4a1c9

Use the output of the cluster list to view the nodes and volumes within the cluster.

$ heketi-cli cluster info 8578ad5529c354e9d21898cba4c4a1c9
Cluster id: 8578ad5529c354e9d21898cba4c4a1c9
Nodes:
8424caa3b7bd9e9098ef200fe8033edf
ca2a1703a3ed68979eef307abe2f1770
ca9344f2390798304b1a7877ecc0bb85
Volumes:
6e59417c9c3c5dec057607e5450dc9ed

OpenShift Container Platform allows for the use of secrets so that items do not need to be stored in clear text. The admin password for heketi, specified during installation with cns-deploy, should be stored in base64-encoding. OCP can refer to this secret instead of specifying the password in clear text.

To generate the base64-encoded equivalent of the admin password supplied to the cns-deploy command perform the following.

$ echo -n myS3cr3tpassw0rd | base64
bXlzZWNyZXRwYXNzdzByZA==

On the master or workstation with the OpenShift client installed and a user with cluster-admin privileges use the base64 password string in the following YAML to define the secret in OpenShift’s default project or namespace.

$ vi heketi-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: heketi-secret
  namespace: default
data:
  key: bXlzZWNyZXRwYXNzdzByZA==
type: kubernetes.io/glusterfs

Create the secret by using the following command.

$ oc create -f heketi-secret.yaml
secret "heketi-secret" created

The cluster-admin or storage-admin can perform the following which will allow for dynamically provisioned CNS storage on demand. The key benefit of this storage is that the persistent storage created can be configured with access modes of ReadWriteOnce(RWO), ReadOnlyMany (ROX), or ReadWriteMany (RWX) adding much more flexibility than cloud provider specific storage.

If Multiple types of CNS storage are desired, additional StorageClass objects can be created to realize multiple tiers of storage defining different types of storage behind a single heketi instance. This will involve deploying more glusterfs pods on additional storage nodes (one gluster pod per OpenShift node) with different type and quality of volumes attached to achieve the desired properties of a tier (e.g. SSDs for “fast” storage, magnetic for “slow” storage). For the examples below we will assume that only one type of storage is required.

Perform the following steps from CLI on a workstation or master node where the OpenShift client has been configured.

$ oc project storage
$ oc get routes heketi
NAME      HOST/PORT     PATH      SERVICES   PORT      TERMINATION   WILDCARD
heketi    heketi-storage.apps.dpl.local             heketi     <all>                   None

$ export HEKETI_CLI_SERVER=http://heketi-storage.apps.dpl.local
$ export HEKETI_CLI_USER=admin
$ export HEKETI_CLI_KEY=myS3cr3tpassw0rd

Record the cluster id of the glusterfs pods in heketi.

$ heketi-cli cluster list
Clusters:
8578ad5529#354e9d21898cba4c4a1c9

The StorageClass object requires both the cluster id and the heketi route to be defined to successfully created. Use the information from the output of heketi-cli cluster list and oc get routes heketi to fill in the resturl and clusterid. For OpenShift 3.4, the value of clusterid is not supported for the StorageClass object. If a value is provided the StorageClass object will fail to create for OpenShift version 3.4. The failure occurs because OpenShift 3.4 can only have a single TSP or CNS cluster.

OpenShift 3.4

$ vi glusterfs-storageclass-slow.yaml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: gluster-cns-slow
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: http://heketi-storage.apps.dpl.local
  restauthenabled: "true"
  restuser: "admin"
  secretNamespace: "default"
  secretName: "heketi-secret"

The StorageClass object can now be created using this yaml file.

$ oc create -f glusterfs-storageclass-slow.yaml

OpenShift 3.6

$ vi glusterfs-storageclass-slow.yaml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: gluster-cns-slow
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: http://heketi-storage.apps.dpl.local
  clusterid: 8578ad5529c354e9d21898cba4c4a1c9
  restauthenabled: "true"
  restuser: "admin"
  secretNamespace: "default"
  secretName: "heketi-secret"

The StorageClass object can now be created using this yaml file.

$ oc create -f glusterfs-storageclass-slow.yaml

To validate the StorageClass object was created perform the following.

$ oc get storageclass gluster-cns-slow
NAME             TYPE
gluster-cns-dd   kubernetes.io/glusterfs
$ oc describe storageclass gluster-cns-slow
Name:		gluster-cns-slow
IsDefaultClass:	No
Annotations:	<none>
Provisioner:	kubernetes.io/glusterfs
Parameters:	clusterid=8578ad5529c354e9d21898cba4c4a1c9,restauthenabled=true,resturl=http://heketi-storage.apps.dpl.local,restuser=admin,secretName=heketi-secret,secretNamespace=default
No events.

To deploy an additional glusterfs pool OCP requires additional nodes to be available that currently are not running glusterfs pods yet. This will require that another three OpenShift nodes are available in the environment using either the add-node with option --node_type=storage script or by manually deploying three instances and installing and configuring those nodes for OpenShift.

Once the new nodes are available, the next step is to get glusterfs pods up and running on the additional nodes. This is achieved by extending the members of the DaemonSet defined in the first CNS deployment. The storagenode=glusterfs label must be applied to the nodes to allow for the scheduling of the glusterfs pods.

First identify the three nodes that will be added to the CNS cluster and then apply the label.

$ oc get nodes --show-labels
NAME                          STATUS                     AGE        VERSION             LABELS
ocp3-app-cns-0.dpl.local      Ready                      8d         v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ocp3-app-cns-0.dpl.local,storagenode=glusterfs
ocp3-app-cns-1.dpl.local      Ready                      8d         v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ocp3-app-cns-1.dpl.local,storagenode=glusterfs
ocp3-app-cns-2.dpl.local      Ready                      8d         v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ocp3-app-cns-2.dpl.local,storagenode=glusterfs
ocp3-app-cns-3.dpl.local      Ready                      1h         v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ocp3-app-cns-3.dpl.local
ocp3-app-cns-4.dpl.local      Ready                      1h         v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ocp3-app-cns-4.dpl.local
ocp3-app-cns-5.dpl.local      Ready                      1h         v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=ocp3-app-cns-5.dpl.local
...omitted...

$ oc label node ocp3-app-cns-3.dpl.local storagenode=glusterfs
$ oc label node ocp3-app-cns-4.dpl.local storagenode=glusterfs
$ oc label node ocp3-app-cns-5.dpl.local storagenode=glusterfs

Once the label has been applied then the glusterfs pods will scale from 3 pods to 6. The glusterfs pods will be running on both the newly labeled nodes and the existing nodes.

$ oc get pods
NAME              READY     STATUS    RESTARTS   AGE
glusterfs-2lcnb   1/1       Running   0          26m
glusterfs-356cf   1/1       Running   0          26m
glusterfs-fh4gm   1/1       Running   0          26m
glusterfs-hg4tk   1/1       Running   0          2m
glusterfs-v759z   1/1       Running   0          1m
glusterfs-x038d   1/1       Running   0          2m
heketi-1-cqjzm    1/1       Running   0          22m

Wait until all of the glusterfs pods are in READY 1/1 state before continuing. The new pods are not yet configured as a CNS cluster. The new glusterfs pods will be a new CNS cluster after the topology.json file is updated to define the new nodes they reside on and the heketi-cli is executed with this new topology.json file as input.

Modify the topology.json file of the first CNS cluster to include a second entry in the “clusters” list containing the additional nodes. The initial nodes have been omitted from the output below but are still required.

$ vi gluster-topology.json
{
  "clusters": [
    {
      "nodes": [
        {
		... nodes from initial cns-deploy ...
        }
      ]
    },
    {
            "nodes": [
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ocp3-app-cns-3"
                            ],
                            "storage": [
                                "172.0.10.214"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/sdd"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ocp3-app-cns-4"
                            ],
                            "storage": [
                                "172.0.10.215"
                            ]
                        },
                        "zone": 2
                    },
                    "devices": [
                        "/dev/sdd"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ocp3-app-cns-5"
                            ],
                            "storage": [
                                "172.0.10.216"
                            ]
                        },
                        "zone": 3
                    },
                    "devices": [
                        "/dev/sdd"
                    ]
                }
            ]
        }
    ]
}

Using heketi-cli load the modified topology.json file via heketi to trigger the creation of a second cluster using the steps below. The first step is to export the values of the heketi server, user, and key. The HEKET_CLI_KEY value should be the same as that created for the first cluster (set using --admin-key for cns-deploy).

$ export HEKETI_CLI_SERVER=http://heketi-storage.apps.dpl.local
$ export HEKETI_CLI_USER=admin
$ export HEKETI_CLI_KEY=myS3cr3tpassw0rd

With these environment variables exported the next step is to load the newly modified topology.json.

$ heketi-cli topology load --json=gluster-topology.json
	Found node ocp3-app-cns-0.dpl.local on cluster ca8d539dbcb480655b611693b2d7b573
		Found device /dev/sdd
	Found node ocp3-app-cns-1.dpl.local on cluster ca8d539dbcb480655b611693b2d7b573
		Found device /dev/sdd
	Found node ocp3-app-cns-2.dpl.local on cluster ca8d539dbcb480655b611693b2d7b573
		Found device /dev/sdd
Creating cluster ... ID: 5cc7333acb8824e4a238217b8f360940
	Creating node ocp3-app-cns-3.dpl.local ... ID: 4a0b77b6fae1ee17ec8a6d72e5e3bf64
		Adding device /dev/sdd ... OK
	Creating node ocp3-app-cns-4.dpl.local ... ID: 6c83714f41913bc686177c14f818d304
		Adding device /dev/sdd ... OK
	Creating node ocp3-app-cns-5.dpl.local ... ID: d151866310b7328c4cfe923317a5d2b1
		Adding device /dev/sdd ... OK

Observe the second cluster being created and verify that there is a new clusterid created in the console output. Verify you now have a second clusterid and that the correct OCP CNS nodes are in the new cluster.

$ heketi-cli cluster list
$ heketi-cli topology info

Create a second StorageClass object via a YAML file similar to the first one with the same heketi route and heketi secret but using the new clusterid and a unique StorageClass object name.

$ vi glusterfs-storageclass-fast.yaml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: gluster-cns-fast
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: http://heketi-storage.apps.dpl.local
  clusterid: 5cc7333acb8824e4a238217b8f360940
  restauthenabled: "true"
  restuser: "admin"
  secretNamespace: "default"
  secretName: "heketi-secret"

Using the OpenShift client create the StorageClass object.

$ oc create -f glusterfs-storageclass-fast.yaml

The second StorageClass object will now be available to make storage requests using gluster-cns-fast when creating the PVC.

$ vi claim2.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: db-fast
 annotations:
   volume.beta.kubernetes.io/storage-class: gluster-cns-fast
spec:
 accessModes:
  - ReadWriteOnce
 resources:
   requests:
     storage: 10Gi

Deployment of Container-Ready Storage (CRS) requires at least 3 virtual machines with at least one unused block storage device or or drive on each node. The virtual machines should have at least 2 CPUs, 32GB RAM, and an unused drive or volume of 100GB or larger per node. An entitlement for Red Hat Gluster Storage is also required to install the Gluster services.

A python script named add-node.py is provided in the openshift-ansible-contrib git repository. When add-node.py is used with the --node_type=storage option the following will be done.

Do the following from the workstation performing the deployment of the OpenShift Reference Architecture. The ocp-on-vmware.ini file used to create the OpenShift deployment must be in the directory where add-node.py is ran from. There are two entries in the ocp-on-vmware.ini file that must be added or modified. They are the vcenter_datastore and container_storage. The VMware datastore entered is where all of the new OpenShift CRS nodes or virtual machines will be stored so verify that this datastore has at least 1TB of available storage.

$ cd /root/git/openshift-ansible-contrib/reference-architecture/vmware-ansible/
$ cat ocp-on-vmware.ini
….omitted….
$ folder/cluster/resource pool in vCenter to organize VMs
vcenter_folder=ocp3
vcenter_datastore=DPLHP380G9-10-SS200-2
vcenter_cluster=OCP3
vcenter_resource_pool=OCP3
vcenter_datacenter=vDPL
….omitted….
$ persistent container storage: none, crs, cns
container_storage=crs

$ ./add-node.py --node_type=storage
Configured inventory values:
	 console_port:  8443
	 deployment_type:  openshift-enterprise
	 openshift_vers:  v3_6
	 vcenter_host:  172.0.10.246
	 vcenter_username:  administrator@vsphere.local
	 vcenter_password: *******
	 vcenter_template_name:  ocp-server-template-2.0.2-new
	 vcenter_folder:  ocp3
	 vcenter_datastore:  DPLHP380G9-10-SS200-2
	 vcenter_cluster:  OCP3
	 vcenter_resource_pool:  OCP3
	 vcenter_datacenter:  vDPL
	 public_hosted_zone:  dpl.local
	 app_dns_prefix:  apps
	 vm_dns:  172.0.10.241
	 vm_gw:  172.0.10.2
	 vm_netmask:  255.255.255.0
	 vm_network:  "Private"
	 rhel_subscription_user: rhn_user
	 rhel_subscription_pass: *******
	 rhel_subscription_server:
	 rhel_subscription_pool:  Red Hat OpenShift Container Platform, Premium*
	 byo_lb:  False
	 lb_host:  haproxy-0
	 byo_nfs:  False
	 nfs_host:  nfs-0
	 nfs_registry_mountpoint:  /exports
	 master_nodes:  3
	 infra_nodes:  3
	 app_nodes:  6
	 storage_nodes:  0
	 vm_ipaddr_start:  172.0.10.201
	 ocp_hostname_prefix:  ocp3-
	 auth_type:  ldap
	 ldap_user:  openshift
	 ldap_user_password: *******
	 ldap_fqdn:  dpl.local
	 openshift_hosted_metrics_deploy:  false
	 openshift_sdn:  redhat/openshift-ovs-subnet
	 containerized:  false
	 container_storage:  crs
	 tag:  None
	 node_number:  1
	 ini_path:  ./ocp-on-vmware.ini
	 node_type:  storage

Continue creating the inventory file with these values? [y/N]: y
Gluster topology file created using /dev/sdd: topology.json
Inventory file created: add-node.json
host_inventory:
  ocp3-crs-0:
    guestname: ocp3-crs-0
    ip4addr: 172.0.10.211
    tag: storage
  ocp3-crs-1:
    guestname: ocp3-crs-1
    ip4addr: 172.0.10.212
    tag: storage
  ocp3-crs-2:
    guestname: ocp3-crs-2
    ip4addr: 172.0.10.213
    tag: storage

Continue adding nodes with these values? [y/N]:
Admin key password for heketi?:
User key password for heketi?:

Both admin-key and user-key are user defined values, they do not exist before this step. The heketi admin key (password) will later be used to create a heketi-secret in OCP. Be sure to note these values as they will be needed in future operations.

CRS requires the instances to use the Red Hat Gluster Storage entitlement which which allows access to the rh-gluster-3-for-rhel-7-server-rpms repository containing the required RPMs for a successful installation.

Ensure the pool that is specified matches a pool available to the RHSM credentials provided (example pool ID shown below).

# subscription-manager register
# subscription-manager attach --pool=8a85f98156981319015699f0183a253c
# subscription-manager repos --disable=''*
# subscription-manager repos --enable=rhel-7-server-rpms
# subscription-manager repos --enable=rh-gluster-3-for-rhel-7-server-rpms

The add-node.py with option --node_type=storage uses iptables and creates the rules needed for Gluster and Heketi on each of the CRS nodes. The following commands can be ran on the 3 new virtual machines if the instances were built without using the script.

# yum -y install firewalld
# systemctl enable firewalld
# systemctl disable iptables
# systemctl stop iptables
# systemctl start firewalld
# firewall-cmd --add-port=24007/tcp --add-port=24008/tcp --add-port=2222/tcp \ --add-port=8080/tcp --add-port=49152-49251/tcp --permanent
# firewall-cmd --reload
# firewall-cmd --list-all

The redhat-storage-server package and dependencies will install all of the required RPMs for a successful Red Hat Gluster Storage installation. Perform the following on the each of the three CRS nodes.

$ yum install -y redhat-storage-server

After successful installation enable and start the glusterd.service.

$ systemctl enable glusterd
$ systemctl start glusterd

Heketi is used to manage the Gluster Trusted Storage Pool(TSP). Heketi is used to perform tasks such as adding volumes, removing volumes, and creating the initial TSP. If the add-node.py with option --node_type=storage script was not used perform the following on the each of the CRS nodes. Heketi can be installed on one of the CRS instances. For the steps below the first CRS Gluster virtual machine will be used.

$ yum install -y heketi heketi-client

Create the heketi private key on the instance designated to run heketi.

$ ssh-keygen -f /etc/heketi/heketi_key -t rsa -N ''
$ chown heketi:heketi /etc/heketi/heketi_key.pub
$ chown heketi:heketi /etc/heketi/heketi_key

Copy the contents of the /etc/heketi/heketi_key.pub into a clipboard and login to each CRS node and paste the contents of the clipboard as a new line into the /root/.ssh/authorized_keys file. Also make sure on each CRS node to modify the /etc/ssh/sshd_config file to allow root passwordless ssh access (enable “PermitRootLogin” and “RSAAuthentication”) and restart sshd.service. This must be done on all 3 instances including the CRS node where the heketi services are running. Also, on each of the 3 virtual machines requiretty must be disabled or removed in /etc/sudoers to allow for management of those hosts using sudo. Ensure that the line below either does not exist in sudoers or that it is commented out.

$ visudo
... omitted ...
#Defaults    requiretty
... omitted ...

On the node where Heketi was installed, edit the /etc/heketi/heketi.json file to setup the SSH executor and the admin and user keys. The heketi admin key (password) will be used to create a heketi-secret in OCP. This secret will then be used during the creation of the StorageClass object.

$ vi /etc/heketi/heketi.json
... omitted ...
"_use_auth": "Enable JWT authorization. Please enable for deployment",
"use_auth": true,

"_jwt": "Private keys for access",
"jwt": {
  "_admin": "Admin has access to all APIs",
  "admin": {
    "key": "myS3cr3tpassw0rd"
  },
  "_user": "User only has access to /volumes endpoint",
  "user": {
    "key": "mys3rs3cr3tpassw0rd"
  }
},

 "glusterfs": {
    "_executor_comment": [
      "Execute plugin. Possible choices: mock, ssh",
      "mock: This setting is used for testing and development.",
      "      It will not send commands to any node.",
      "ssh:  This setting will notify Heketi to ssh to the nodes.",
      "      It will need the values in sshexec to be configured.",
      "kubernetes: Communicate with GlusterFS containers over",
      "            Kubernetes exec api."
    ],
    "executor": "ssh",
    "_sshexec_comment": "SSH username and private key file information",
    "sshexec": {
      "keyfile": "/etc/heketi/heketi_key",
      "user": "root",
      "port": "22",
      "fstab": "/etc/fstab"
    },
... omitted ...

Restart and enable heketi service to use the configured /etc/heketi/heketi.json file.

$ systemctl restart heketi
$ systemctl enable heketi

The heketi service should now be running. Heketi provides an endpoint to perform a health check. This validation can be done from either an OCP master or from any of the CRS nodes. The hostname is the CRS node where Heketi is installed, in this case ocp3-crs-0.dpl.local.

$ curl http://ocp3-crs-0.dpl.local:8080/hello
Hello from Heketi

The topology.json is used to tell heketi about the environment and which nodes and storage devices it will manage. There is a sample file located in /usr/share/heketi/topology-sample.json and an example shown below for 3 CRS nodes on 3 different VMware hypervisors. Both CRS and CNS use the same format for the topology.json file.

$ vi topology.json
{
    "clusters": [
        {
            "nodes": [
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ocp3-crs-0.dpl.local"
                            ],
                            "storage": [
                                "172.0.10.211"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/sdd"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ocp3-crs-1.dpl.local"
                            ],
                            "storage": [
                                "172.0.10.212"
                            ]
                        },
                        "zone": 2
                    },
                    "devices": [
                        "/dev/sdd"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ocp3-crs-2.dpl.local"
                            ],
                            "storage": [
                                "172.0.10.213"
                            ]
                        },
                        "zone": 3
                    },
                    "devices": [
                        "/dev/sdd"
                    ]
                }
            ]
        }
    ]
}

The HEKETI_CLI_SERVER, HEKETI_CLI_USER, and HEKETI_CLI_KEY environment variables are required for heketi-cli commands to be ran. The HEKETI_CLI_SERVER is the CRS node name where the heketi services are running. The HEKETI_CLI_KEY is the admin key value configured in the /etc/heketi/heketi.json file.

$ export HEKETI_CLI_SERVER=http://ocp3-crs-0.dpl.local:8080
$ export HEKETI_CLI_USER=admin
$ export HEKETI_CLI_KEY=myS3cr3tpassw0rd

Using heketi-cli, run the following command to load the topology of your environment.

$ heketi-cli topology load --json=topology.json
      Creating cluster ... ID: bb802020a9c2c5df45f42075412c8c05
	Creating node ocp3-crs-0.dpl.local ... ID: b45d38a349218b8a0bab7123e004264b
				Adding device /dev/sdd ... OK
	Creating node ocp3-crs-1.dpl.local ... ID: 2b3b30efdbc3855a115d7eb8fdc800fe
				Adding device /dev/sdd ... OK
	Creating node ocp3-crs-2.dpl.local ... ID: c7d366ae7bd61f4b69de7143873e8999
				Adding device /dev/sdd ... OK

OCP allows for the use of secrets so that items do not need to be stored in clear text. The admin password for heketi, specified during configuration of the heketi.json file, should be stored in base64-encoding. OCP can refer to this secret instead of specifying the password in clear text.

To generate the base64-encoded equivalent of the admin password supplied to the cns-deploy command perform the following.

$ echo -n myS3cr3tpassw0rd | base64
bXlTM2NyM3RwYXNzdzByZA==

On the master or workstation with the OCP client installed with cluster-admin privileges use the base64 password string in the following YAML to define the secret in OCP’s default namespace.

$ vi heketi-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: heketi-secret
  namespace: default
data:
  key: bXlTM2NyM3RwYXNzdzByZA==
type: kubernetes.io/glusterfs

Create the secret by using the following command.

$ oc create -f heketi-secret.yaml
secret "heketi-secret" created

CRS storage has all of the same benefits that CNS storage has with regard to OpenShift storage. The cluster-admin or storage-admin can perform the following which will allow for dynamically provisioned CRS storage on demand. The key benefit of this storage is that the persistent storage can be created with access modes ReadWriteOnce(RWO), ReadOnlyMany(ROX), or ReadWriteMany(RWX).

A StorageClass object requires certain parameters to be defined to successfully create the resource. Use the values of the exported environment variables from the previous steps to define the resturl, restuser, secretNamespace, and secretName.

$ vi storageclass.yaml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: crs-gluster
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://ocp3-crs-0.dpl.local:8080"
  restauthenabled: "true"
  restuser: "admin"
  secretNamespace: "default"
  secretName: "heketi-secret"

Once the Storage Class json file has been created use the oc create command to create the object in OpenShift.

$ oc create -f storageclass.yaml

To validate the Storage Class was created perform the following.

$ oc get storageclass
NAME             TYPE
crs-gluster kubernetes.io/glusterfs

$ oc describe storageclass crs-gluster
Name:		crs-gluster
IsDefaultClass:	No
Annotations:	<none>
Provisioner:	kubernetes.io/glusterfs
Parameters:	restauthenabled=true,resturl=http://ocp3-crs-0.dpl.local:8080,restuser=admin,secretName=heketi-secret,secretNamespace=default
No events.

The Storage Class created in the previous section allows for storage to be dynamically provisioned using the CRS resources. The example below shows a dynamically provisioned volume being requested from the crs-gluster StorageClass object. A sample persistent volume claim is provided below:

$ vi db-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: db
 annotations:
   volume.beta.kubernetes.io/storage-class: crs-gluster
spec:
 accessModes:
  - ReadWriteOnce
 resources:
   requests:
     storage: 10Gi

$ oc create -f db-claim.yaml
persistentvolumeclaim "db" created