An existing port group and virtual LAN (VLAN) are required for deployment. The environment can utilize a virtual dedicated server (vDS) or vSwitch. The specifics of that are unimportant. However, to utilize network IO control and some of the quality of service (QoS) technologies that VMware employs, a vDS is required.

The vSphere hosts should have shared storage for the VMware virtual machine disk files (VMDKs). A best practice recommendation is to enable storage I/O control (SIOC) to address any performance issues caused by latency. This article discusses in depth how to do this.

OpenShift Container Platform can be configured to access VMware vSphere VMDK Volumes, including using VMware vSphere VMDK Volumes as persistent storage for application data.

The vSphere Cloud Provider allows using vSphere managed storage within OpenShift Container Platform and supports:

$ curl -LO https://github.com/vmware/govmomi/releases/download/v0.15.0/govc_linux_amd64.gz
$ gunzip govc_linux_amd64.gz
$ chmod +x govc_linux_amd64
$ cp govc_linux_amd64 /usr/bin/govc

$ export GOVC_URL='vCenter IP OR FQDN'
$ export GOVC_USERNAME='vCenter User'
$ export GOVC_PASSWORD='vCenter Password'
$ export GOVC_INSECURE=1

$ govc ls /<datacenter>/vm/<vm-folder-name>

for each VM in $(govc ls /<datacenter>/vm/<vm-folder-name>);do govc vm.change -e="disk.enableUUID=1" -vm="$VM"

[Global]
        user = "username" 1
        password = "password" 2
        server = "10.10.0.2" 3
        port = "443" 4
        insecure-flag = "1" 5
        datacenter = "datacenter-name" 6
        datastore = "datastore-name" 7
        working-dir = "vm-folder-path" 8
        vm-uuid = "vm-uuid" 9
[Disk]
        scsicontrollertype = pvscsi
        network = "VM Network" 10

During the installation of Red Hat OpenShift Container Platform, the VMware instances created for RHOCP should include various VMDK volumes to ensure various OpenShift directories do not fill up the disk or cause disk contention in the /var partition.

Container images are stored locally on the nodes running Red Hat OpenShift Container Platform pods. The container-storage-setup script uses the /etc/sysconfig/docker-storage-setup file to specify the storage configuration.

The /etc/sysconfig/docker-storage-setup file must be created before starting the docker service, otherwise the storage is configured using a loopback device. The container storage setup is performed on all hosts running containers, therefore masters, infrastructure, and application nodes.

# cat /etc/sysconfig/docker-storage-setup
DEVS="/dev/sdb"
VG="docker-vol"
DATA_SIZE="95%VG"
STORAGE_DRIVER=overlay2
CONTAINER_ROOT_LV_NAME="dockerlv"
CONTAINER_ROOT_LV_MOUNT_PATH="/var/lib/docker"

A VMDK volume should be created on the master instances for the storage of /var/lib/etcd. Storing etcd allows the similar benefit of protecting /var but more importantly provides the ability to perform snapshots of the volume when performing etcd maintenance.

A VMDK volume should be created for the directory of /var/lib/origin/openshift.local.volumes that is used with the perFSGroup setting at installation and with the mount option of gquota. These settings and volumes set a quota to ensure that containers cannot grow to an unreasonable size.

# mkfs -t xfs /dev/sdc
# vi /etc/fstab
/dev/mapper/rhel-root   /                       xfs     defaults        0 0
UUID=8665acc0-22ee-4e45-970c-ae20c70656ef /boot                   xfs     defaults        0 0
/dev/sdc /var/lib/origin/openshift.local.volumes xfs gquota 0 0

Red Hat Enterprise Linux 7 is the only OS supported by the Red Hat OpenShift Container Platform installer therefore provider infrastructure deployment and installer must be run from one of the following locations:

This reference architecture focuses on deploying and installing Red Hat OpenShift Container Platform from local workstation/server/virtual machine. Jenkins CI/CD and Bastion are out of scope.

This reference environment should consist of the following instances:

The installation process for Red Hat OpenShift Container Platform depends on a reliable name service that contains an address record for each of the target instances.

An example DNS configuration is listed below:

$ORIGIN apps.example.com.
*           A       10.x.y.200
$ORIGIN example.com.
haproxy-0       A   10.x.y.200
infra-0         A   10.x.y.100
infra-1         A   10.x.y.101
infra-2         A   10.x.y.102
master-0        A   10.x.y.103
master-1        A   10.x.y.104
master-2        A   10.x.y.105
app-0           A   10.x.y.106
app-1           A   10.x.y.107
app-2           A   10.x.y.108

$ govc ls /<datacenter>/vm/<folder>/

$ ssh master-1

If an organization currently does not have a load balancer in place then HAProxy can be deployed. A load balancer such as HAProxy provides a single view of the Red Hat OpenShift Container Platform master services for the applications. The master services and the applications use different TCP ports so a single TCP load balancer can handle all of the inbound connections.

The load balanced DNS name that developers use must be in a DNS A record pointing to the haproxy server before installation. For applications, a wildcard DNS entry must point to the haproxy host.

The configuration of the HAProxy instance is completed within the subsequent steps as the deployment host configures the Red Hat subscriptions for all the instances and the Red Hat OpenShift Container Platform installer auto configures the HAProxy instance based upon the information found within the Red Hat OpenShift Container Platform inventory file.

Ensure connectivity to all instances via the deployment instance via:

$ ansible all -m ping

Once connectivity to all instances has been established, register the instances via Red Hat Subscription Manager. This is accomplished using credentials or an activation key.

Via credentials the ansible command is as follows:

$ ansible all -m command -a "subscription-manager register --username <user> --password '<password>'"

Via activation key, the ansible command is as follows:

$ ansible all -m command -a "subscription-manager register --org=<org_id> --activationkey=<keyname>"

where the following options:

Once all the instances have been successfully registered, enable all the required RHOCP repositories on all the instances via:

$ ansible all -m command -a "subscription-manager repos \
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-extras-rpms" \
    --enable="rhel-7-server-ose-3.9-rpms" \
    --enable="rhel-7-fast-datapath-rpms" \
    --enable="rhel-7-server-ansible-2.4-rpms""

Red Hat OpenShift Container Platform provides the ability to use many different authentication platforms. For this reference architecture, LDAP is the preferred authentication mechanism. A listing of other authentication options are available at Configuring Authentication and User Agent.

When configuring LDAP as the authentication provider the following parameters can be added to the ansible inventory. An example is shown below.

openshift_master_identity_providers=[{'name': 'idm', 'challenge': 'true', 'login': 'true', 'kind': 'LDAPPasswordIdentityProvider', 'attributes': {'id': ['dn'], 'email': ['mail'], 'name': ['cn'], 'preferredUsername': ['uid']}, 'bindDN': 'uid=admin,cn=users,cn=accounts,dc=openshift,dc=com', 'bindPassword': 'ldapadmin', 'ca': '/etc/origin/master/ca.crt', 'insecure': 'false', 'url': 'ldap://ldap.example.com/cn=users,cn=accounts,dc=openshift,dc=com?uid?sub?(memberOf=cn=ose-user,cn=groups,cn=accounts,dc=openshift,dc=com)'}]

It can be useful to check for potential issues or misconfigurations in the instances before continuing the installation process. Connect to every instance using the deployment host and verify the disks are properly created and mounted.

$ ssh deployment.example.com
$ ssh <instance>
$ lsblk
$ sudo journalctl
$ free -m
$ sudo yum repolist

where instance is for example master-0.example.com

For reference, below is example output of lsblk for the master nodes.

$ lsblk
NAME                     MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                        8:0    0   60G  0 disk
├─sda1                     8:1    0  500M  0 part /boot
├─sda2                     8:2    0 39.5G  0 part
│ ├─rhel-root            253:0    0   55G  0 lvm  /
│ └─rhel-swap            253:1    0  3.9G  0 lvm
└─sda3                     8:3    0   20G  0 part
  └─rhel-root            253:0    0   55G  0 lvm  /
sdb                        8:16   0   40G  0 disk
└─sdb1                     8:17   0   40G  0 part
  └─docker--vol-dockerlv 253:2    0   40G  0 lvm  /var/lib/docker
sdc                        8:32   0   40G  0 disk /var/lib/origin/openshift.local.volumes
sdd                        8:48   0   40G  0 disk /var/lib/etcd

For reference, below is an example of output of lsblk for the infra and app nodes.

$ lsblk
sda                        8:0    0   60G  0 disk
├─sda1                     8:1    0  500M  0 part /boot
├─sda2                     8:2    0 39.5G  0 part
│ ├─rhel-root            253:0    0   55G  0 lvm  /
│ └─rhel-swap            253:1    0  3.9G  0 lvm
└─sda3                     8:3    0   20G  0 part
  └─rhel-root            253:0    0   55G  0 lvm  /
sdb                        8:16   0   40G  0 disk
└─sdb1                     8:17   0   40G  0 part
  └─docker--vol-dockerlv 253:2    0   40G  0 lvm  /var/lib/docker
sdc                        8:32   0   40G  0 disk /var/lib/origin/openshift.local.volumes
sdd                        8:48   0  300G  0 disk

Prior to Chapter 4, Operational Management, create DRS anti-affinity rules to ensure maximum availability for the cluster.

The following VMware documentation goes over creating and configuring anti-affinity rules in depth.

Lastly, set all of the VMs created to High VM Latency to ensure some additional tuning recommended by VMware for latency sensitive workloads as described here.

Figure 2.1. VMware High Latency

The Red Hat OpenShift Container Platform Ansible installation provides a playbook to ensure all prerequisites are met prior to the installation of Red Hat OpenShift Container Platform. This includes steps such as registering all the nodes with Red Hat Subscription Manager and setting up the docker on the docker volumes.

Via the ansible-playbook command on the deployment instance, ensure all the prerequisites are met using prerequisites.yml playbook:

$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml

In the event that OpenShift fails to install or the prerequisites playbook fails, follow the steps in Appendix Appendix F, Troubleshooting Ansible by Red Hat to troubleshoot Ansible.