Within this reference environment, the instances are deployed in a single Azure Region which can be selected when running the ARM template. Although the default region can be changed, the reference architecture deployment should only be used in regions with premium storage for performance reasons.

All VMs are created using the On-Demand Red Hat Enterprise Linux (RHEL) image and the size used by default is Standard_DS4_v2 for masters and nodes and Standard_DS1_v2 for the bastion host. Instance sizing can be changed when the ARM template is run which is covered in later chapters.

$ lsblk
NAME                              MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0                                 2:0    1    4K  0 disk
sda                                 8:0    0   32G  0 disk
├─sda1                              8:1    0  500M  0 part /boot
└─sda2                              8:2    0 31,5G  0 part /
sdb                                 8:16   0   28G  0 disk
└─sdb1                              8:17   0   28G  0 part /mnt/resource
sdc                                 8:32   0  128G  0 disk
└─sdc1                              8:33   0  128G  0 part /var/lib/origin/openshift.local.volumes
sdd                                 8:48   0  128G  0 disk
└─sdd1                              8:49   0  128G  0 part
  ├─docker--vg-docker--pool_tmeta 253:0    0  132M  0 lvm
  │ └─docker--vg-docker--pool     253:2    0   51G  0 lvm
  └─docker--vg-docker--pool_tdata 253:1    0   51G  0 lvm
    └─docker--vg-docker--pool     253:2    0   51G  0 lvm
sr0                                11:0    1  1,1M  0 rom

Two Azure Load Balancers (ALB) are used in this reference environment. The table below describes the ALB, the load balancer DNS name, the instances in which the Azure Load Balancers (ALB) is attached, and the port monitored by the load balancer to state whether an instance is in or out of service.

The External load balancer utilizes the OpenShift Container Platform master API port for communication internally and externally. The Router load balancer uses the public subnets and maps to infrastructure nodes. The infrastructure nodes run the router pod which then directs traffic directly from the outside world into pods when external routes are defined.

To avoid reconfiguring DNS every time a new route is created, an external wildcard A DNS entry record must be configured pointing to the Router load balancer IP.

For example, create a wildcard DNS entry for cloudapps.example.com that has a low time-to-live value (TTL) and points to the public IP address of the Router load balancer:

*.cloudapps.example.com. 300 IN A 192.168.133.2

The following tables provide the installed software versions for the different servers that make up the Red Hat OpenShift Container Platform highly available reference environment.

A subscription to the following channels is required in order to deploy this reference environment’s configuration.

The subscriptions are accessed via a pool id, which is a required parameter in the ARM template that will deploy the VMs in the Microsoft Azure environment and it is located in the reference-architecture/azure-ansible/azuredeploy.parameters.json file in the openshift-ansible-contrib repository

This section describes the environment and setup needed to execute the ARM template and perform post installation tasks.

In order to deploy the environment from the template, a Microsoft Azure subscription is required. A trial subscription is not recommended, as the reference architecture uses significant resources, and the typical trial subscription does not provide adequate resources.

The deployment of OpenShift Container Platform requires a user that has the proper permissions by the Microsoft Azure administrator. The user must be able to create accounts, storage accounts, roles, policies, load balancers, and deploy virtual machine instances. It is helpful to have delete permissions in order to be able to redeploy the environment while testing.

An OpenShift Container Platform cluster is deployed with-in one Azure Region. In order to get the best possible availability in Microsoft Azure, availability sets are implemented.

In Microsoft Azure, virtual machines (VMs) can be placed in to a logical grouping called an availability set. When creating VMs within an availability set, the Microsoft Azure platform distributes the placement of those VMs across the underlying infrastructure. Should there be a planned maintenance event to the Microsoft Azure platform or an underlying hardware/infrastructure fault, the use of availability sets ensures that at least one VM remains running. The Microsoft Azure SLA requires two or more VMs within an availability set to allow the distribution of VMs across the underlying infrastructure.

SSH keys are used instead of passwords in the OpenShift Container Platform installation process. These keys are generated on the system that will be used to login and manage the system. In addition, they are automatically distributed by the ARM template to all virtual machines that are created.

In order to use the template, SSH public and private keys are needed. To avoid asking for the passphrase, do not not apply a passphrase to the key.

The public key will be injected in the ~/.ssh/authorized_keys file in all the hosts, and the private key will be copied to the ~/.ssh/id_rsa file in all the hosts to allow SSH communication within the environment (i.e.- from the bastion to master1 without passwords).

$ ssh-keygen -t rsa -N '' -f /home/USER/.ssh/id_rsa

Your identification has been saved in /home/USER/.ssh/id_rsa.
Your public key has been saved in /home/USER/.ssh/id_rsa.pub.
The key fingerprint is:
e7:97:c7:e2:0e:f9:0e:fc:c4:d7:cb:e5:31:11:92:14 USER@sysdeseng.rdu.redhat.com
The key's randomart image is:
+--[ RSA 2048]----+
|             E.  |
|            . .  |
|             o . |
|              . .|
|        S .    . |
|         + o o ..|
|          * * +oo|
|           O +..=|
|           o*  o.|
+-----------------+

In the Microsoft Azure environment, resources such as storage accounts, virtual networks and virtual machines (VMs) are grouped together in resource groups as a single entity and their names must be unique to an Microsoft Azure subscription. Note that multiple resource groups are supported in a region, as well as having the same resource group in multiple regions but a resource group may not span resources in multiple regions.

An Azure VNet provides the ability to set up custom virtual networking which includes subnets, and IP address ranges. In this reference implementation guide, a dedicated VNet is created with all its accompanying services to provide a stable network for the OpenShift Container Platform deployment.

A VNet is created as a logical representation of a networking environment in the Microsoft Azure cloud. The following subnets and CIDR listed below are used.

The VNet is created and a human readable tag is assigned. Three subnets are created in the VNet. The bastion instance is on the Master Subnet. The two internal load balancers allow access to the OpenShift Container Platform API and console and the routing of application traffic. All the VMs are able to communicate to the internet for packages, container images, and external git repositories.

OpenShift Container Platform uses a software-defined networking (SDN) approach to provide a unified cluster network that enables communication between pods across the OpenShift Container Platform cluster. This pod network is established and maintained by the OpenShift SDN, which configures an overlay network using Open vSwitch (OVS).

There are three different plug-ins available in OpenShift Container Platform 3.5 for configuring the pod network:

The plugin used in this reference architecture can be specified among the supported ones at deployment time using the ARM template. The default value is redhat/ovs-multitenant that allows multitenant isolation for pods per project.

For more information about OpenShift Container Platform networking, see OpenShift SDN documentation.

The purpose of the Microsoft Azure Network security groups (NSG) is to restrict traffic from outside of the VNet to servers inside of the VNet. The Network security groups also are used to restrict server to server communications inside the VNet. Network security groups provide an extra layer of security similar to a firewall: in the event a port is opened on an instance, the security group will not allow the communication to the port unless explicitly stated in a Network security group.

NSG are grouped depending on the traffic flow (inbound or outbound) and every NSG contains rules where every rule specify:

NSG rules are processed by priority meaning the first rule matching the traffic it is applied.

All the security groups contains default rules to block connectivity coming from outside the VNet, where the default rules allow and disallow traffic as follows:

Once the Network security group is created, it should be associated to an infrastructure component where using the resource manager mechanism to deploy infrastructure in Microsoft Azure they can be associated to a NIC or a subnet.

The Network security groups are specified on each node type json file, located in https://github.com/openshift/openshift-ansible-contrib/tree/master/reference-architecture/azure-ansible/ (like master.json for master instances)

DNS is an integral part of a successful OpenShift Container Platform environment. Microsoft Azure provides a DNS-as-a-Service called Azure DNS, per Microsoft; "The Microsoft global network of name servers has the scale and redundancy to ensure ultra-high availability for your domains. With Microsoft Azure DNS, you can be sure that your DNS will always be available."

Microsoft Azure provides the DNS for public zone, as well as internal host resolution. These are configured automatically during the execution of the reference architecture scripts.

$ cat /etc/resolv.conf
# Generated by NetworkManager
search fesj5eh114uebnc5jfpnxi33kh.dx.internal.cloudapp.net
nameserver 168.63.129.16

$ nslookup master1
Server:		168.63.129.16
Address:	168.63.129.16#53

Name:	master1.fesj5eh114uebnc5jfpnxi33kh.dx.internal.cloudapp.net
Address: 10.0.0.5

The bastion server implements mainly two distinct functions. One is that of a secure way to connect to all the nodes, and second that of the "installer" of the system. The information provided to the ARM template is passed to the bastion host and from there, playbooks and scripts are automatically generated and executed, resulting in OpenShift Container Platform being installed.

As shown in the Figure 2.1, “bastion diagram” the bastion server in this reference architecture provides a secure way to limit SSH access to the Microsoft Azure environment. The master and node security groups only allow for SSH connectivity between nodes inside of the Security Group while the bastion allows SSH access from everywhere. The bastion host is the only ingress point for SSH in the cluster from external entities. When connecting to the OpenShift Container Platform infrastructure, the bastion forwards the request to the appropriate server.

Figure 2.1. bastion diagram

Ansible relies on inventory files and variables to perform playbook runs. As part of the reference architecture provided Ansible playbooks, the inventory is created during the boot of the bastion host. The Azure Resource Manager templates (ARM) passes parameters via a script extension to RHEL on the bastion. On the bastion host a bastion.sh script generates the inventory file in /etc/ansible/hosts.

[OSEv3:children]
masters
nodes
etcd
new_nodes
new_masters

[OSEv3:vars]
osm_controller_args={'cloud-provider': ['azure'], 'cloud-config': ['/etc/azure/azure.conf']}
osm_api_server_args={'cloud-provider': ['azure'], 'cloud-config': ['/etc/azure/azure.conf']}
openshift_node_kubelet_args={'cloud-provider': ['azure'], 'cloud-config': ['/etc/azure/azure.conf'], 'enable-controller-attach-detach': ['true']}
debug_level=2
console_port=8443
docker_udev_workaround=True
openshift_node_debug_level="{{ node_debug_level | default(debug_level, true) }}"
openshift_master_debug_level="{{ master_debug_level | default(debug_level, true) }}"
openshift_master_access_token_max_seconds=2419200
openshift_hosted_router_replicas=3
openshift_hosted_registry_replicas=3
openshift_master_api_port="{{ console_port }}"
openshift_master_console_port="{{ console_port }}"
openshift_override_hostname_check=true
osm_use_cockpit=false
openshift_release=v3.5
openshift_cloudprovider_kind=azure
openshift_node_local_quota_per_fsgroup=512Mi
azure_resource_group=${RESOURCEGROUP}
rhn_pool_id=${RHNPOOLID}
openshift_install_examples=true
deployment_type=openshift-enterprise
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
openshift_master_manage_htpasswd=false

os_sdn_network_plugin_name=${OPENSHIFTSDN}

# default selectors for router and registry services
openshift_router_selector='role=infra'
openshift_registry_selector='role=infra'

# Select default nodes for projects
osm_default_node_selector="role=app"
ansible_become=yes
ansible_ssh_user=${AUSERNAME}
remote_user=${AUSERNAME}

openshift_master_default_subdomain=${WILDCARDNIP}
osm_default_subdomain=${WILDCARDNIP}
openshift_use_dnsmasq=true
openshift_public_hostname=${RESOURCEGROUP}.${FULLDOMAIN}

openshift_master_cluster_method=native
openshift_master_cluster_hostname=${RESOURCEGROUP}.${FULLDOMAIN}
openshift_master_cluster_public_hostname=${RESOURCEGROUP}.${FULLDOMAIN}

openshift_metrics_install_metrics=false
openshift_metrics_cassandra_storage_type=pv
openshift_metrics_cassandra_pvc_size="${METRICS_CASSANDRASIZE}G"
openshift_metrics_cassandra_replicas="${METRICS_INSTANCES}"
openshift_metrics_hawkular_nodeselector={"role":"infra"}
openshift_metrics_cassandra_nodeselector={"role":"infra"}
openshift_metrics_heapster_nodeselector={"role":"infra"}

openshift_logging_install_logging=false
openshift_logging_es_pv_selector={"usage":"elasticsearch"}
openshift_logging_es_pvc_dynamic="false"
openshift_logging_es_pvc_size="${LOGGING_ES_SIZE}G"
openshift_logging_es_cluster_size=${LOGGING_ES_INSTANCES}
openshift_logging_fluentd_nodeselector={"logging":"true"}
openshift_logging_es_nodeselector={"role":"infra"}
openshift_logging_kibana_nodeselector={"role":"infra"}
openshift_logging_curator_nodeselector={"role":"infra"}

openshift_logging_use_ops=false
openshift_logging_es_ops_pv_selector={"usage":"opselasticsearch"}
openshift_logging_es_ops_pvc_dynamic="false"
openshift_logging_es_ops_pvc_size="${OPSLOGGING_ES_SIZE}G"
openshift_logging_es_ops_cluster_size=${OPSLOGGING_ES_INSTANCES}
openshift_logging_es_ops_nodeselector={"role":"infra"}
openshift_logging_kibana_ops_nodeselector={"role":"infra"}
openshift_logging_curator_ops_nodeselector={"role":"infra"}

[masters]
master1 openshift_hostname=master1 openshift_node_labels="{'role': 'master'}"
master2 openshift_hostname=master2 openshift_node_labels="{'role': 'master'}"
master3 openshift_hostname=master3 openshift_node_labels="{'role': 'master'}"

[etcd]
master1
master2
master3

[new_nodes]
[new_masters]

[nodes]
master1 openshift_hostname=master1 openshift_node_labels="{'role':'master','zone':'default','logging':'true'}" openshift_schedulable=false
master2 openshift_hostname=master2 openshift_node_labels="{'role':'master','zone':'default','logging':'true'}" openshift_schedulable=false
master3 openshift_hostname=master3 openshift_node_labels="{'role':'master','zone':'default','logging':'true'}" openshift_schedulable=false
infranode1 openshift_hostname=infranode1 openshift_node_labels="{'role': 'infra', 'zone': 'default','logging':'true'}"
infranode2 openshift_hostname=infranode2 openshift_node_labels="{'role': 'infra', 'zone': 'default','logging':'true'}"
infranode3 openshift_hostname=infranode3 openshift_node_labels="{'role': 'infra', 'zone': 'default','logging':'true'}"
node[01:${NODECOUNT}] openshift_hostname=node[01:${NODECOUNT}] openshift_node_labels="{'role':'app','zone':'default','logging':'true'}"

For the OpenShift Container Platform installation, the ARM template collects the needed parameters, creates the virtual machines, and passes the parameters to the virtual machines, where a node type specific script in bash will take the parameters and generate the needed playbooks and automation. During this process each VM is assigned an ansible tag, that allows the playbooks to address the different node types.

Nodes are Microsoft Azure instances that serve a specific purpose for OpenShift Container Platform. OpenShift Container Platform masters are also configured as nodes as they are part of the SDN. Nodes deployed on Microsoft Azure can be vertically scaled before or after the OpenShift Container Platform installation using the Microsoft Azure dashboard. There are three types of nodes as described below.

OpenShift Container Platform leverages the Kubernetes concept of a pod, which is one or more containers deployed together on one host, and the smallest compute unit that can be defined, deployed, and managed.

A pod could be just a single container that runs a php application connecting to a database outside of the OpenShift Container Platform environment, or a pod could be two containers, one of them runs a php application and the other one runs an ephemeral database. Pods have the ability to be scaled at runtime or at the time of launch using the OpenShift Container Platform web console, the OpenShift Container Platform API or the oc CLI tool.

For more information about the OpenShift Container Platform architecture and components, see the OpenShift Architecture

Pods inside of an OpenShift Container Platform cluster are only reachable via their IP addresses on the cluster network. To be able to access pods from outside the OpenShift Container Platform cluster, OpenShift Container Platform provides a few options:

OpenShift Container Platform routers provide external hostname mapping and load balancing to services exposed by users over protocols that pass distinguishing information directly to the router; the hostname must be present in the protocol in order for the router to determine where to send it. Routers currently support the following protocols:

There are two different router plug-ins that can be deployed in OpenShift Container Platform:

The HAProxy template router enable routes created by developers to be used by external clients. To avoid reconfiguration of the DNS servers every time a route is created, the suggested method is to define a wildcard DNS entry that will redirect every hostname to the router.

OpenShift Container Platform can build container images from source code, deploy them, and manage their lifecycle. To enable this it is required to deploy the Integrated OpenShift Container Platform Registry

The registry stores container images and metadata. For production environment, persistent storage is recommended to be used by the registry, otherwise any images built or pushed into the registry would disappear if the pod were to restart.

The registry is scaled to 3 pods/instances to allow for HA, and load balancing. The default load balancing settings use the source ip address to enforce session stickiness. Failure of a pod may result in a pull or push operation to fail, but the operation may be restarted. The failed registry pod will be automatically restarted.

Using the installation methods described in this document the registry is deployed using Azure Blob Storage, an Microsoft Azure service that provides object storage. In order to use Azure Blob Storage the registry configuration has been extended. The procedure used is detailed in the OpenShift and Docker documentation, and it is to modify the registry deploymentconfig to add the Azure Blob Storage service details as:

$ oc env dc docker-registry -e REGISTRY_STORAGE=azure -e REGISTRY_STORAGE_AZURE_ACCOUNTNAME=<azure_storage_account_name> -e REGISTRY_STORAGE_AZURE_ACCOUNTKEY=<azure_storage_account_key> -e REGISTRY_STORAGE_AZURE_CONTAINER=registry

This will be done automatically as part of the installation by the scripts provided in the git repository.

There are several options when it comes to authentication of users in OpenShift Container Platform. OpenShift Container Platform can leverage an existing identity provider within an organization such as LDAP or can use external identity providers like GitHub, Google, and GitLab. The configuration of identification providers occurs on the OpenShift Container Platform master instances and multiple identity providers can be specified. The reference architecture document uses htpasswd as the authentication provider but any of the other mechanisms would be an acceptable choice. Roles can be customized and added to user accounts or groups to allow for extra privileges such as the ability to list nodes or assign persistent storage volumes to a project.

For the use cases considered in this reference architecture, including OpenShift Container Platform applications that connect to containerized databases or need some basic persistent storage, we need to consider multiple storage solutions in the cloud and different architectural approaches. Microsoft Azure offers a number of storage choices that offer high durability with three simultaneous replicas, including Standard and Premium storage.

Furthermore, a use case requirement is to implement "shared storage" where the volume should allow simultaneous read and write operations. Upon reviewing multiple options supported by OpenShift Container Platform and the underlying Red Hat Enterprise Linux infrastructure, a choice was made to use the Azure VHD based storage server to give the Microsoft Azure OpenShift Container Platform nodes the best match of performance and flexibility, and use Azure Blob Storage for the registry storage.

Red Hat OpenShift Container Platform environments can be enriched by deploying an optional component named Red Hat OpenShift Container Platform metrics that collect metrics exposed by the kubelet from pods running in the environment and provides the ability to view CPU, memory, and network-based metrics and display the values in the user interface.

Red Hat OpenShift Container Platform metrics it is composed by a few pods running on the Red Hat OpenShift Container Platform environment:

It is important to understand capacity planning when deploying metrics into an OpenShift environment regarding that one set of metrics pods (Cassandra/Hawkular/Heapster) is able to monitor at least 25,000 pods.

Red Hat OpenShift Container Platform metrics components can be customized for longer data persistence, pods limits, replicas of individual components, custom certificates, etc.

Within this reference environment, metrics are deployed optionally on the infrastructure nodes depending on the "metrics" parameter of the ARM template. When "true" is selected, it deploys one set of metric pods (Cassandra/Hawkular/Heapster) on the infrastructure nodes (to avoid using resources on the application nodes) and uses persistent storage to allow for metrics data to be preserved for 7 days.

One of the Red Hat OpenShift Container Platform optional components named Red Hat OpenShift Container Platform aggregated logging collects and aggregates logs for a range of Red Hat OpenShift Container Platform services enabling Red Hat OpenShift Container Platform users view the logs of projects which they have view access using a web interface.

Red Hat OpenShift Container Platform aggregated logging component it is a modified version of the ELK stack composed by a few pods running on the Red Hat OpenShift Container Platform environment:

Once deployed in a cluster, the stack aggregates logs from all nodes and projects into Elasticsearch, and provides a Kibana UI to view any logs. Cluster administrators can view all logs, but application developers can only view logs for projects they have permission to view. To avoid users to see logs from pods in other projects, the Search guard plugin for Elasticsearch is used.

A separate Elasticsearch cluster, a separate Kibana, and a separate Curator components can be deployed to form the OPS cluster where logs for the default, openshift, and openshift-infra projects as well as /var/log/messages on nodes are automatically aggregated and grouped into the .operations item in the Kibana interface.

Red Hat OpenShift Container Platform aggregated logging components can be customized for longer data persistence, pods limits, replicas of individual components, custom certificates, etc.

Within this reference environment, aggregated logging components are deployed optionally on the infrastructure nodes depending on the "logging" parameter of the ARM template. When "true" is selected, it deploys on the infrastructure nodes (to avoid using resources on the application nodes) the following elements:

Also, there is an "opslogging" parameter that can optionally deploy the same architecture but for operational logs: