Within this reference environment, the instances are deployed in a single Azure Region which can be selected when running the ARM template. Although the default region can be changed, the reference architecture deployment should only be used in regions with premium storage for performance reasons.

All VMs are created using the On-Demand Red Hat Enterprise Linux (RHEL) image and the size used by default is Standard_DS4_v2 for masters and nodes and Standard_DS1_v2 for the bastion host. Instance sizing can be changed when the ARM template is run which is covered in later chapters.

$ lsblk
NAME                                                                                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
fd0                                                                                            2:0    1     4K  0 disk
sda                                                                                            8:0    0   128G  0 disk
├─sda1                                                                                         8:1    0   500M  0 part /boot
└─sda2                                                                                         8:2    0 127.5G  0 part /
sdb                                                                                            8:16   0    56G  0 disk
└─sdb1                                                                                         8:17   0    56G  0 part /mnt/resource
sdc                                                                                            8:32   0   128G  0 disk
└─sdc1                                                                                         8:33   0   128G  0 part /var/lib/origin/openshift.local.volumes
sdd                                                                                            8:48   0   128G  0 disk
└─sdd1                                                                                         8:49   0   128G  0 part
  ├─docker--vg-docker--pool_tmeta                                                            253:0    0   132M  0 lvm
  │ └─docker--vg-docker--pool                                                                253:2    0 121.6G  0 lvm

sed -i -e 's/ResourceDisk.EnableSwap.*/ResourceDisk.EnableSwap=n/g' /etc/waagent.conf

sed -i -e 's/ResourceDisk.SwapSizeMB.*/ResourceDisk.SwapSizeMB=0/g' /etc/waagent.conf

swapoff -a

Two Azure Load Balancers (ALB) are used in this reference environment. The table below describes the ALB, the load balancer DNS name, the instances in which the Azure Load Balancers (ALB) is attached, and the port monitored by the load balancer to state whether an instance is in or out of service.

The External load balancer utilizes the OpenShift Container Platform master API port for communication internally and externally. The Router load balancer uses the public subnets and maps to infrastructure nodes. The infrastructure nodes run the router pod which then directs traffic directly from the outside world into pods when external routes are defined.

To avoid reconfiguring DNS every time a new route is created, an external wildcard A DNS entry record must be configured pointing to the Router load balancer IP.

For example, create a wildcard DNS entry for cloudapps.example.com that has a low time-to-live value (TTL) and points to the public IP address of the Router load balancer:

*.cloudapps.example.com. 300 IN A 192.168.133.2

The following tables provide the installed software versions for the different servers that make up the Red Hat OpenShift Container Platform highly available reference environment.

A subscription to the following channels is required in order to deploy this reference environment’s configuration.

The subscriptions are accessed via a pool id, which is a required parameter in the ARM template that will deploy the VMs in the Microsoft Azure environment and it is located in the reference-architecture/azure-ansible/3.6/azuredeploy.parameters.json file in the openshift-ansible-contrib repository

This section describes the environment and setup needed to execute the ARM template and perform post installation tasks.

In order to deploy the environment from the template, a Microsoft Azure subscription is required. A trial subscription is not recommended, as the reference architecture uses significant resources, and the typical trial subscription does not provide adequate resources.

The deployment of OpenShift Container Platform requires a user that has the proper permissions by the Microsoft Azure administrator. The user must be able to create accounts, storage accounts, roles, policies, load balancers, and deploy virtual machine instances. It is helpful to have delete permissions in order to be able to redeploy the environment while testing.

An OpenShift Container Platform cluster is deployed with-in one Azure Region. In order to get the best possible availability in Microsoft Azure, availability sets are implemented.

In Microsoft Azure, virtual machines (VMs) can be placed in to a logical grouping called an availability set. When creating VMs within an availability set, the Microsoft Azure platform distributes the placement of those VMs across the underlying infrastructure. Should there be a planned maintenance event to the Microsoft Azure platform or an underlying hardware/infrastructure fault, the use of availability sets ensures that at least one VM remains running. The Microsoft Azure SLA requires two or more VMs within an availability set to allow the distribution of VMs across the underlying infrastructure.

SSH keys are used instead of passwords in the OpenShift Container Platform installation process. These keys are generated on the system that will be used to login and manage the system. In addition, they are automatically distributed by the ARM template to all virtual machines that are created.

In order to use the template, SSH public and private keys are needed. To avoid asking for the passphrase, do not not apply a passphrase to the key.

The public key will be injected in the ~/.ssh/authorized_keys file in all the hosts, and the private key will be copied to the ~/.ssh/id_rsa file in all the hosts to allow SSH communication within the environment (i.e.- from the bastion to master1 without passwords).

$ ssh-keygen -t rsa -N '' -f /home/USER/.ssh/id_rsa

Your identification has been saved in /home/USER/.ssh/id_rsa.
Your public key has been saved in /home/USER/.ssh/id_rsa.pub.
The key fingerprint is:
e7:97:c7:e2:0e:f9:0e:fc:c4:d7:cb:e5:31:11:92:14 USER@sysdeseng.rdu.redhat.com
The key's randomart image is:
+--[ RSA 2048]----+
|             E.  |
|            . .  |
|             o . |
|              . .|
|        S .    . |
|         + o o ..|
|          * * +oo|
|           O +..=|
|           o*  o.|
+-----------------+

In the Microsoft Azure environment, resources such as storage accounts, virtual networks and virtual machines (VMs) are grouped together in resource groups as a single entity and their names must be unique to an Microsoft Azure subscription. Note that multiple resource groups are supported in a region, as well as having the same resource group in multiple regions but a resource group may not span resources in multiple regions.

An Azure VNet provides the ability to set up custom virtual networking which includes subnets, and IP address ranges. In this reference implementation guide, a dedicated VNet is created with all its accompanying services to provide a stable network for the OpenShift Container Platform deployment.

A VNet is created as a logical representation of a networking environment in the Microsoft Azure cloud. The following subnets and CIDR listed below are used.

The VNet is created and a human readable tag is assigned. Three subnets are created in the VNet. The bastion instance is on the Master Subnet. The two internal load balancers allow access to the OpenShift Container Platform API and console and the routing of application traffic. All the VMs are able to communicate to the internet for packages, container images, and external git repositories.

OpenShift Container Platform uses a software-defined networking (SDN) approach to provide a unified cluster network that enables communication between pods across the OpenShift Container Platform cluster. This pod network is established and maintained by the OpenShift SDN, which configures an overlay network using Open vSwitch (OVS).

There are three different plug-ins available in OpenShift Container Platform 3.6 for configuring the pod network:

The plugin used in this reference architecture can be specified among the supported ones at deployment time using the ARM template. The default value is redhat/ovs-multitenant that allows multitenant isolation for pods per project.

For more information about OpenShift Container Platform networking, see OpenShift SDN documentation.

The purpose of the Microsoft Azure Network security groups (NSG) is to restrict traffic from outside of the VNet to servers inside of the VNet. The Network security groups also are used to restrict server to server communications inside the VNet. Network security groups provide an extra layer of security similar to a firewall: in the event a port is opened on an instance, the security group will not allow the communication to the port unless explicitly stated in a Network security group.

NSG are grouped depending on the traffic flow (inbound or outbound) and every NSG contains rules where every rule specify:

NSG rules are processed by priority meaning the first rule matching the traffic it is applied.

All the security groups contains default rules to block connectivity coming from outside the VNet, where the default rules allow and disallow traffic as follows:

Once the Network security group is created, it should be associated to an infrastructure component where using the resource manager mechanism to deploy infrastructure in Microsoft Azure they can be associated to a NIC or a subnet.

The Network security groups are specified on each node type json file, located in https://github.com/openshift/openshift-ansible-contrib/tree/master/reference-architecture/azure-ansible/3.6/ (such as master.json for master instances)

DNS is an integral part of a successful OpenShift Container Platform environment. Microsoft Azure provides a DNS-as-a-Service called Azure DNS, per Microsoft; "The Microsoft global network of name servers has the scale and redundancy to ensure ultra-high availability for your domains. With Microsoft Azure DNS, you can be sure that your DNS will always be available."

Microsoft Azure provides the DNS for public zone, as well as internal host resolution. These are configured automatically during the execution of the reference architecture scripts.

$ cat /etc/resolv.conf
# Generated by NetworkManager
search fesj5eh114uebnc5jfpnxi33kh.dx.internal.cloudapp.net
nameserver 168.63.129.16

$ nslookup master1
Server:		168.63.129.16
Address:	168.63.129.16#53

Name:	master1.fesj5eh114uebnc5jfpnxi33kh.dx.internal.cloudapp.net
Address: 10.0.0.5

The bastion server implements mainly two distinct functions. One is that of a secure way to connect to all the nodes, and second that of the "installer" of the system. The information provided to the ARM template is passed to the bastion host and from there, playbooks and scripts are automatically generated and executed, resulting in OpenShift Container Platform being installed.

As shown in the Figure 2.1, “bastion diagram” the bastion server in this reference architecture provides a secure way to limit SSH access to the Microsoft Azure environment. The master and node security groups only allow for SSH connectivity between nodes inside of the Security Group while the bastion allows SSH access from everywhere. The bastion host is the only ingress point for SSH in the cluster from external entities. When connecting to the OpenShift Container Platform infrastructure, the bastion forwards the request to the appropriate server.

Figure 2.1. bastion diagram

Ansible relies on inventory files and variables to perform playbook runs. As part of the reference architecture provided Ansible playbooks, the inventory is created during the boot of the bastion host. The Azure Resource Manager templates (ARM) passes parameters via a script extension to RHEL on the bastion. On the bastion host a bastion.sh script generates the inventory file in /etc/ansible/hosts.

[OSEv3:children]
masters
nodes
etcd
new_nodes
new_masters

[OSEv3:vars]
osm_controller_args={'cloud-provider': ['azure'], 'cloud-config': ['/etc/azure/azure.conf']}
osm_api_server_args={'cloud-provider': ['azure'], 'cloud-config': ['/etc/azure/azure.conf']}
openshift_node_kubelet_args={'cloud-provider': ['azure'], 'cloud-config': ['/etc/azure/azure.conf'], 'enable-controller-attach-detach': ['true']}
debug_level=2
console_port=8443
docker_udev_workaround=True
openshift_node_debug_level="{{ node_debug_level | default(debug_level, true) }}"
openshift_master_debug_level="{{ master_debug_level | default(debug_level, true) }}"
openshift_master_access_token_max_seconds=2419200
openshift_hosted_router_replicas=3
openshift_hosted_registry_replicas=3
openshift_master_api_port="{{ console_port }}"
openshift_master_console_port="{{ console_port }}"
openshift_override_hostname_check=true
osm_use_cockpit=false
openshift_release=v3.6
openshift_cloudprovider_kind=azure
openshift_node_local_quota_per_fsgroup=512Mi
azure_resource_group=${RESOURCEGROUP}
rhn_pool_id=${RHNPOOLID}
openshift_install_examples=true
deployment_type=openshift-enterprise
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
openshift_master_manage_htpasswd=false

os_sdn_network_plugin_name=${OPENSHIFTSDN}

# default selectors for router and registry services
openshift_router_selector='role=infra'
openshift_registry_selector='role=infra'

# Select default nodes for projects
osm_default_node_selector="role=app"
ansible_become=yes
ansible_ssh_user=${AUSERNAME}
remote_user=${AUSERNAME}

openshift_master_default_subdomain=${WILDCARDNIP}
osm_default_subdomain=${WILDCARDNIP}
openshift_use_dnsmasq=true
openshift_public_hostname=${RESOURCEGROUP}.${FULLDOMAIN}

openshift_master_cluster_method=native
openshift_master_cluster_hostname=${RESOURCEGROUP}.${FULLDOMAIN}
openshift_master_cluster_public_hostname=${RESOURCEGROUP}.${FULLDOMAIN}

openshift_metrics_install_metrics=false
openshift_metrics_cassandra_storage_type=pv
openshift_metrics_cassandra_pvc_size="${METRICS_CASSANDRASIZE}G"
openshift_metrics_cassandra_replicas="${METRICS_INSTANCES}"
openshift_metrics_hawkular_nodeselector={"role":"infra"}
openshift_metrics_cassandra_nodeselector={"role":"infra"}
openshift_metrics_heapster_nodeselector={"role":"infra"}

openshift_logging_install_logging=false
openshift_logging_es_pv_selector={"usage":"elasticsearch"}
openshift_logging_es_pvc_dynamic="false"
openshift_logging_es_pvc_size="${LOGGING_ES_SIZE}G"
openshift_logging_es_cluster_size=${LOGGING_ES_INSTANCES}
openshift_logging_fluentd_nodeselector={"logging":"true"}
openshift_logging_es_nodeselector={"role":"infra"}
openshift_logging_kibana_nodeselector={"role":"infra"}
openshift_logging_curator_nodeselector={"role":"infra"}

openshift_logging_use_ops=false
openshift_logging_es_ops_pv_selector={"usage":"opselasticsearch"}
openshift_logging_es_ops_pvc_dynamic="false"
openshift_logging_es_ops_pvc_size="${OPSLOGGING_ES_SIZE}G"
openshift_logging_es_ops_cluster_size=${OPSLOGGING_ES_INSTANCES}
openshift_logging_es_ops_nodeselector={"role":"infra"}
openshift_logging_kibana_ops_nodeselector={"role":"infra"}
openshift_logging_curator_ops_nodeselector={"role":"infra"}

[masters]
master1 openshift_hostname=master1 openshift_node_labels="{'role': 'master'}"
master2 openshift_hostname=master2 openshift_node_labels="{'role': 'master'}"
master3 openshift_hostname=master3 openshift_node_labels="{'role': 'master'}"

[etcd]
master1
master2
master3

[new_nodes]
[new_masters]

[nodes]
master1 openshift_hostname=master1 openshift_node_labels="{'role':'master','zone':'default','logging':'true'}" openshift_schedulable=false
master2 openshift_hostname=master2 openshift_node_labels="{'role':'master','zone':'default','logging':'true'}" openshift_schedulable=false
master3 openshift_hostname=master3 openshift_node_labels="{'role':'master','zone':'default','logging':'true'}" openshift_schedulable=false
infranode1 openshift_hostname=infranode1 openshift_node_labels="{'role': 'infra', 'zone': 'default','logging':'true'}"
infranode2 openshift_hostname=infranode2 openshift_node_labels="{'role': 'infra', 'zone': 'default','logging':'true'}"
infranode3 openshift_hostname=infranode3 openshift_node_labels="{'role': 'infra', 'zone': 'default','logging':'true'}"
node[01:${NODECOUNT}] openshift_hostname=node[01:${NODECOUNT}] openshift_node_labels="{'role':'app','zone':'default','logging':'true'}"

For the OpenShift Container Platform installation, the ARM template collects the needed parameters, creates the virtual machines, and passes the parameters to the virtual machines, where a node type specific script in bash will take the parameters and generate the needed playbooks and automation. During this process each VM is assigned an ansible tag, that allows the playbooks to address the different node types.

Nodes are Microsoft Azure instances that serve a specific purpose for OpenShift Container Platform. OpenShift Container Platform masters are also configured as nodes as they are part of the SDN. Nodes deployed on Microsoft Azure can be vertically scaled before or after the OpenShift Container Platform installation using the Microsoft Azure dashboard. There are three types of nodes as described below.

OpenShift Container Platform leverages the Kubernetes concept of a pod, which is one or more containers deployed together on one host, and the smallest compute unit that can be defined, deployed, and managed.

A pod could be just a single container that runs a php application connecting to a database outside of the OpenShift Container Platform environment, or a pod could be two containers, one of them runs a php application and the other one runs an ephemeral database. Pods have the ability to be scaled at runtime or at the time of launch using the OpenShift Container Platform web console, the OpenShift Container Platform API or the oc CLI tool.

For more information about the OpenShift Container Platform architecture and components, see the OpenShift Architecture

Pods inside of an OpenShift Container Platform cluster are only reachable via their IP addresses on the cluster network. To be able to access pods from outside the OpenShift Container Platform cluster, OpenShift Container Platform provides a few options:

OpenShift Container Platform routers provide external hostname mapping and load balancing to services exposed by users over protocols that pass distinguishing information directly to the router; the hostname must be present in the protocol in order for the router to determine where to send it. Routers currently support the following protocols:

There are two different router plug-ins that can be deployed in OpenShift Container Platform:

The HAProxy template router enable routes created by developers to be used by external clients. To avoid reconfiguration of the DNS servers every time a route is created, the suggested method is to define a wildcard DNS entry that will redirect every hostname to the router.

OpenShift Container Platform can build container images from source code, deploy them, and manage their lifecycle. To enable this it is required to deploy the Integrated OpenShift Container Platform Registry

The registry stores container images and metadata. For production environment, persistent storage is recommended to be used by the registry, otherwise any images built or pushed into the registry would disappear if the pod were to restart.

The registry is scaled to 3 pods/instances to allow for HA, and load balancing. The default load balancing settings use the source ip address to enforce session stickiness. Failure of a pod may result in a pull or push operation to fail, but the operation may be restarted. The failed registry pod will be automatically restarted.

Using the installation methods described in this document the registry is deployed using Azure Blob Storage, an Microsoft Azure service that provides object storage. In order to use Azure Blob Storage the registry configuration has been extended. The procedure used is detailed in the OpenShift and Docker documentation, and it is to modify the registry deploymentconfig to add the Azure Blob Storage service details as:

$ oc env dc docker-registry -e REGISTRY_STORAGE=azure -e REGISTRY_STORAGE_AZURE_ACCOUNTNAME=<azure_storage_account_name> -e REGISTRY_STORAGE_AZURE_ACCOUNTKEY=<azure_storage_account_key> -e REGISTRY_STORAGE_AZURE_CONTAINER=registry

This will be done automatically as part of the installation by the scripts provided in the git repository.

There are several options when it comes to authentication of users in OpenShift Container Platform. OpenShift Container Platform can leverage an existing identity provider within an organization such as LDAP or can use external identity providers like GitHub, Google, and GitLab. The configuration of identification providers occurs on the OpenShift Container Platform master instances and multiple identity providers can be specified. The reference architecture document uses htpasswd and sso as the authentication providers but any of the other mechanisms would be an acceptable choice. Roles can be customized and added to user accounts or groups to allow for extra privileges such as the ability to list nodes or assign persistent storage volumes to a project.

For the use cases considered in this reference architecture, including OpenShift Container Platform applications that connect to containerized databases or need some basic persistent storage, we need to consider multiple storage solutions in the cloud and different architectural approaches. Microsoft Azure offers a number of storage choices that offer high durability with three simultaneous replicas, including Standard and Premium storage.

Furthermore, a use case requirement is to implement "shared storage" where the volume should allow simultaneous read and write operations. Upon reviewing multiple options supported by OpenShift Container Platform and the underlying Red Hat Enterprise Linux infrastructure, a choice was made to use the Azure VHD based storage server to give the Microsoft Azure OpenShift Container Platform nodes the best match of performance and flexibility, and use Azure Blob Storage for the registry storage.

Red Hat OpenShift Container Platform environments can be enriched by deploying an optional component named Red Hat OpenShift Container Platform metrics that collect metrics exposed by the kubelet from pods running in the environment and provides the ability to view CPU, memory, and network-based metrics and display the values in the user interface.

Red Hat OpenShift Container Platform metrics it is composed by a few pods running on the Red Hat OpenShift Container Platform environment:

It is important to understand capacity planning when deploying metrics into an OpenShift environment regarding that one set of metrics pods (Cassandra/Hawkular/Heapster) is able to monitor at least 25,000 pods.

Red Hat OpenShift Container Platform metrics components can be customized for longer data persistence, pods limits, replicas of individual components, custom certificates, etc.

Within this reference environment, metrics are deployed optionally on the infrastructure nodes depending on the "metrics" parameter of the ARM template. When "true" is selected, it deploys one set of metric pods (Cassandra/Hawkular/Heapster) on the infrastructure nodes (to avoid using resources on the application nodes) and uses persistent storage to allow for metrics data to be preserved for 7 days.

One of the Red Hat OpenShift Container Platform optional components named Red Hat OpenShift Container Platform aggregated logging collects and aggregates logs for a range of Red Hat OpenShift Container Platform services enabling Red Hat OpenShift Container Platform users view the logs of projects which they have view access using a web interface.

Red Hat OpenShift Container Platform aggregated logging component it is a modified version of the ELK stack composed by a few pods running on the Red Hat OpenShift Container Platform environment:

Once deployed in a cluster, the stack aggregates logs from all nodes and projects into Elasticsearch, and provides a Kibana UI to view any logs. Cluster administrators can view all logs, but application developers can only view logs for projects they have permission to view. To avoid users to see logs from pods in other projects, the Search guard plugin for Elasticsearch is used.

A separate Elasticsearch cluster, a separate Kibana, and a separate Curator components can be deployed to form the OPS cluster where logs for the default, openshift, and openshift-infra projects as well as /var/log/messages on nodes are automatically aggregated and grouped into the .operations item in the Kibana interface.

Red Hat OpenShift Container Platform aggregated logging components can be customized for longer data persistence, pods limits, replicas of individual components, custom certificates, etc.

Within this reference environment, aggregated logging components are deployed optionally on the infrastructure nodes depending on the "logging" parameter of the ARM template. When "true" is selected, it deploys on the infrastructure nodes (to avoid using resources on the application nodes) the following elements:

Also, there is an "opslogging" parameter that can optionally deploy the same architecture but for operational logs:

The script and playbooks provided within the git repository deploys infrastructure, installs and configures OpenShift Container Platform, and performs post installation tasks such as scaling the router and registry. The playbooks create specific roles, policies, and users required for cloud provider configuration in OpenShift Container Platform, as well as storage accounts on Microsoft Azure. In order to deploy OpenShift Container Platform on Microsoft Azure, an ARM template must be created. The ARM template takes the basic information, admin user name, password, Red Hat Subscription Manager username and password, SSH public and private keys, as well as offering the ability to size the virtual machines.

$ cat ~/.ssh/id_rsa | base64 | tr -d '\n' | xclip -selection clipboard

$ cat ~/.ssh/id_rsa | base64 | tr -d '\n' | pbcopy

# subscription-manager list --available

+-------------------------------------------+
    Available Subscriptions
+-------------------------------------------+
Subscription Name:   Red Hat OpenShift Container Platform, Premium (1-2 Sockets)
Provides:            Red Hat OpenShift Enterprise Application Node
                     Red Hat OpenShift Container Platform
                     Red Hat OpenShift Enterprise Client Tools
                     Red Hat Enterprise Linux Server
                     Red Hat Enterprise Linux Atomic Host
                     Red Hat OpenShift Enterprise Infrastructure
                     Red Hat CloudForms
SKU:                 MCT2862
Contract:            38154141
Pool ID:             1b152951269116311123bc522341e1
Provides Management: No
Available:           64
Suggested:           1
Service Level:       Premium
Service Type:        L1-L3
Subscription Type:   Stackable
Ends:                25/08/2017
System Type:         Virtual

$ subscription-manager identity
system identity: 8c85ab6d-495c-451e-b38e-5f09bc3342a0
name: bastion
org name: 6616399
org ID: 6616399

Azure Resource Manager templates consist of json files that describes the objects that will be deployed in Microsoft Azure. The main template file for this reference architecture is located in the reference-architecture/azure-ansible/3.6/azuredeploy.json file in the git repository. This file is the main ARM template that launches all the other templates under azure-ansible. There are four types of virtual machines created by the template (bastion, master node, infrastructure node and application node) and for each of these types there is a additional json file that defines each VM type.

The ARM template for each type, automatically starts a bash shell script that does part of the initial setup. The main shell script is the reference-architecture/azure-ansible/3.6/bastion.sh that handles the generation of the ansible host inventory, as well as the setup and running of ansible across all the hosts. The bastion host also provides isolation of all the hosts in the resource group from the public internet for the purpose of SSH access.

In addition to the production template of azuredeploy.json, a single virtual machine version is also available. This template is located at: reference-architecture/azure-ansible/3.6/allinone.json This provides for early prototypes and tests of applications in a ocp environment. Note that the single VM does not support the high-availability and load-balancing features of the full azuredeploy.json template. The single virtual machine template only allows you to choose the vm size, and uses a single public ip for console and applications. The Number of Nodes and Wildcard Zone are removed.

In order to provision the OpenShift Container Platform environment using the ARM template, the following information is required:

There are two ways to provision the OpenShift Container Platform environment. Using a Web Interface, by filling out a form generated by the template for the needed parameters. And the alternate way, is using a ansible playbook to deploy the cluster. The ansible method is ideal when you wish to deploy clusters in a repeatable way, or when you wish to have more than one cluster.

3.5.1. Provisioning ARM Template by using the Web Interface

With the above information ready, go to https://github.com/openshift/openshift-ansible-contrib/tree/master/reference-architecture/azure-ansible and click the [Deploy To Azure] button in the "OCP Version 3.6 - Create the Installation on the Azure Portal" section. This shows the form, to allow the deployment to be started.

Figure 3.1. ARM Template

$ sudo yum -y install ansible git

$ git clone https://github.com/openshift/openshift-ansible-contrib
$ cd openshift-ansible-contrib/reference-architecture/azure-ansible/3.6/ansibledeployocp/

$ ansible-playbook playbooks/prepare.yml

[default]
subscription_id=00000000-0000-0000-0000-000000000000
tenant=11111111-1111-1111-1111-111111111111
client_id=33333333-3333-3333-3333-333333333
secret=ServicePrincipalPassword

$ sudo yum install -y nodejs
$ sudo npm install -g azure-cli
$ azure login
$ azure account show
info:    Executing command account show
data:    Name                        : Acme Inc.
data:    ID                          : 00000000-0000-0000-0000-000000000000
data:    State                       : Enabled
data:    Tenant ID                   : 11111111-1111-1111-1111-111111111111
data:    Is Default                  : true
data:    Environment                 : AzureCloud
data:    Has Certificate             : Yes
data:    Has Access Token            : Yes
data:    User name                   : youremail@yourcompany.com
data:
info:    account show command OK

$ azure ad sp create -n azureansible -p ServicePrincipalPassword

info:    Executing command ad sp create
+ Creating application ansiblelab
+ Creating service principal for application 33333333-3333-3333-3333-333333333
data:    Object Id:               44444444-4444-4444-4444-444444444444
data:    Display Name:            azureansible
data:    Service Principal Names:
data:                             33333333-3333-3333-3333-333333333
data:                             http://azureansible
info:    ad sp create command OK

$ cp vars.yaml.example vars.yaml
$ vim vars.yaml

$ ansible-playbook -e @vars.yaml playbooks/deploy.yaml

PLAY [localhost] 

TASK [Destroy Azure Deploy]  changed: [localhost] TASK [Destroy Azure Deploy] 
ok: [localhost]

TASK [Create Azure Deploy] 
changed: [localhost]

PLAY RECAP **
localhost                  : ok=3    changed=2    unreachable=0    failed=0

Once the playbooks have successfully completed the next steps will be to perform the steps defined in Chapter 4, Operational Management. In the event that OpenShift Container Platform failed to install, follow the steps in Appendix C: Chapter 9, Installation Failure to restart the installation of OpenShift Container Platform.

The OpenShift Registry Console deployment is deployed on any Red Hat OpenShift Container Platform node by default, so the container may end up running on any of the application nodes.

From the first master instance(master1), ensure the OpenShift Registry Console pod runs on the infra nodes by modifying the nodeSelector as follows:

$ *ssh <user>@<resourcegroup>b.<region>.cloudapp.azure.com*
$ *ssh <user>@master1*
$ oc patch dc registry-console \
    -n default \
    -p '{"spec":{"template":{"spec":{"nodeSelector":{"role":"infra"}}}}}'

At this point the infrastructure and Red Hat OpenShift Container Platform have been deployed. Log into the Azure web console and check the resources. In the Azure web console, check for the following resources:

After the Azure Resource Manager template is submitted, and the ARM deployment succeeds, the ansible install is started automatically.

A wildcard DNS entry must be created if a custom domain is required, by default the nip.io service is be used as explained in the Microsoft Azure DNS section.

Optionally, to be able to connect easily to the VMs, the following SSH configuration file can be applied to the workstation that will perform the SSH commands:

$ cat /home/<user>/.ssh/config

Host bastion
     HostName                 <resourcegroup>b.<region>.cloudapp.azure.com
     User                     <user>
     StrictHostKeyChecking    no
     ProxyCommand             none
     CheckHostIP              no
     ForwardAgent             yes
     IdentityFile             /home/<user>/.ssh/id_rsa

Host master? infranode? node??
     ProxyCommand             ssh <user>@bastion -W %h:%p
     user                     <user>
     IdentityFile             /home/<user>/.ssh/id_rsa

To connect to any VM it is only needed the hostname as:

$ ssh infranode3

With all of the steps that occur during the installation of OpenShift Container Platform, it is possible to lose track of the names of the instances in the recently deployed environment. One option to get these hostnames is to browse to the Azure Resource Group dashboard and select Overview. The filter shows all instances relating to the reference architecture deployment.

To help facilitate the Chapter 4, Operational Management chapter the following hostnames will be used.

To run diagnostics, SSH into the first master node (master1), via the bastion host using the admin user specified in the template:

$ ssh <user>@<resourcegroup>b.<region>.cloudapp.azure.com
$ ssh <user>@master1
$ sudo -i

Connectivity to the first master node (master1.<region>.cloudapp.azure.com) as the root user should have been established. Run the diagnostics that are included as part of the OpenShift Container Platform installation:

# oadm diagnostics
[Note] Determining if client configuration exists for client/cluster diagnostics
Info:  Successfully read a client config file at '/root/.kube/config'
Info:  Using context for cluster-admin access: 'default/sysdeseng-westus-cloudapp-azure-com:8443/system:admin'
[Note] Performing systemd discovery

[Note] Running diagnostic: ConfigContexts[default/sysdeseng-westus-cloudapp-azure-com:8443/system:admin]
       Description: Validate client config context is complete and has connectivity

Info:  The current client config context is 'default/sysdeseng-westus-cloudapp-azure-com:8443/system:admin':
       The server URL is 'https://sysdeseng.westus.cloudapp.azure.com:8443'
       The user authentication is 'system:admin/sysdeseng-westus-cloudapp-azure-com:8443'
       The current project is 'default'
       Successfully requested project list; has access to project(s):
         [default gsw kube-system logging management-infra openshift openshift-infra]

[Note] Running diagnostic: DiagnosticPod
       Description: Create a pod to run diagnostics from the application standpoint

       [Note] Running diagnostic: PodCheckDns
              Description: Check that DNS within a pod works as expected

       [Note] Summary of diagnostics execution (version v3.6.5.5):
       [Note] Warnings seen: 0
       [Note] Errors seen: 0

[Note] Running diagnostic: NetworkCheck
       Description: Create a pod on all schedulable nodes and run network diagnostics from the application standpoint

       [Note] Running diagnostic: CheckExternalNetwork
              Description: Check that external network is accessible within a pod

       [Note] Running diagnostic: CheckNodeNetwork
              Description: Check that pods in the cluster can access its own node.

       [Note] Running diagnostic: CheckPodNetwork
              Description: Check pod to pod communication in the cluster. In case of ovs-subnet network plugin, all pods should be able to communicate with each other and in case of multitenant network plugin, pods in non-global projects should be isolated and pods in global projects should be able to access any pod in the cluster and vice versa.

       [Note] Running diagnostic: CheckServiceNetwork
              Description: Check pod to service communication in the cluster. In case of ovs-subnet network plugin, all pods should be able to communicate with all services and in case of multitenant network plugin, services in non-global projects should be isolated and pods in global projects should be able to access any service in the cluster.

       [Note] Running diagnostic: CollectNetworkInfo
              Description: Collect network information in the cluster.

       [Note] Summary of diagnostics execution (version v3.6.5.5):
       [Note] Warnings seen: 0


       [Note] Running diagnostic: CheckNodeNetwork
              Description: Check that pods in the cluster can access its own node.

       [Note] Running diagnostic: CheckPodNetwork
              Description: Check pod to pod communication in the cluster. In case of ovs-subnet network plugin, all pods should be able to communicate with each other and in case of multitenant network plugin, pods in non-global projects should be isolated and pods in global projects should be able to access any pod in the cluster and vice versa.

       [Note] Running diagnostic: CheckServiceNetwork
              Description: Check pod to service communication in the cluster. In case of ovs-subnet network plugin, all pods should be able to communicate with all services and in case of multitenant network plugin, services in non-global projects should be isolated and pods in global projects should be able to access any service in the cluster.

       [Note] Running diagnostic: CollectNetworkInfo
              Description: Collect network information in the cluster.

       [Note] Summary of diagnostics execution (version v3.6.5.5):
       [Note] Warnings seen: 0


       [Note] Running diagnostic: CheckNodeNetwork
              Description: Check that pods in the cluster can access its own node.

       [Note] Running diagnostic: CheckPodNetwork
              Description: Check pod to pod communication in the cluster. In case of ovs-subnet network plugin, all pods should be able to communicate with each other and in case of multitenant network plugin, pods in non-global projects should be isolated and pods in global projects should be able to access any pod in the cluster and vice versa.

       [Note] Running diagnostic: CheckServiceNetwork
              Description: Check pod to service communication in the cluster. In case of ovs-subnet network plugin, all pods should be able to communicate with all services and in case of multitenant network plugin, services in non-global projects should be isolated and pods in global projects should be able to access any service in the cluster.

       [Note] Running diagnostic: CollectNetworkInfo
              Description: Collect network information in the cluster.

       [Note] Summary of diagnostics execution (version v3.6.5.5):
       [Note] Warnings seen: 0

[Note] Skipping diagnostic: AggregatedLogging
       Description: Check aggregated logging integration for proper configuration
       Because: No LoggingPublicURL is defined in the master configuration

[Note] Running diagnostic: ClusterRegistry
       Description: Check that there is a working Docker registry

[Note] Running diagnostic: ClusterRoleBindings
       Description: Check that the default ClusterRoleBindings are present and contain the expected subjects

Info:  clusterrolebinding/cluster-readers has more subjects than expected.

       Use the oadm policy reconcile-cluster-role-bindings command to update the role binding to remove extra subjects.

Info:  clusterrolebinding/cluster-readers has extra subject {ServiceAccount management-infra management-admin    }.
Info:  clusterrolebinding/cluster-readers has extra subject {ServiceAccount default router    }.

Info:  clusterrolebinding/self-provisioners has more subjects than expected.

       Use the oadm policy reconcile-cluster-role-bindings command to update the role binding to remove extra subjects.

Info:  clusterrolebinding/self-provisioners has extra subject {ServiceAccount management-infra management-admin    }.

[Note] Running diagnostic: ClusterRoles
       Description: Check that the default ClusterRoles are present and contain the expected permissions

[Note] Running diagnostic: ClusterRouterName
       Description: Check there is a working router

[Note] Running diagnostic: MasterNode
       Description: Check if master is also running node (for Open vSwitch)

WARN:  [DClu3004 from diagnostic MasterNode@openshift/origin/pkg/diagnostics/cluster/master_node.go:164]
       Unable to find a node matching the cluster server IP.
       This may indicate the master is not also running a node, and is unable
       to proxy to pods over the Open vSwitch SDN.

[Note] Skipping diagnostic: MetricsApiProxy
       Description: Check the integrated heapster metrics can be reached via the API proxy
       Because: The heapster service does not exist in the openshift-infra project at this time,
       so it is not available for the Horizontal Pod Autoscaler to use as a source of metrics.

[Note] Running diagnostic: NodeDefinitions
       Description: Check node records on master

WARN:  [DClu0003 from diagnostic NodeDefinition@openshift/origin/pkg/diagnostics/cluster/node_definitions.go:112]
       Node master1 is ready but is marked Unschedulable.
       This is usually set manually for administrative reasons.
       An administrator can mark the node schedulable with:
           oadm manage-node master1 --schedulable=true

       While in this state, pods should not be scheduled to deploy on the node.
       Existing pods will continue to run until completed or evacuated (see
       other options for 'oadm manage-node').

WARN:  [DClu0003 from diagnostic NodeDefinition@openshift/origin/pkg/diagnostics/cluster/node_definitions.go:112]
       Node master2 is ready but is marked Unschedulable.
       This is usually set manually for administrative reasons.
       An administrator can mark the node schedulable with:
           oadm manage-node master2 --schedulable=true

       While in this state, pods should not be scheduled to deploy on the node.
       Existing pods will continue to run until completed or evacuated (see
       other options for 'oadm manage-node').

WARN:  [DClu0003 from diagnostic NodeDefinition@openshift/origin/pkg/diagnostics/cluster/node_definitions.go:112]
       Node master3 is ready but is marked Unschedulable.
       This is usually set manually for administrative reasons.
       An administrator can mark the node schedulable with:
           oadm manage-node master3 --schedulable=true

       While in this state, pods should not be scheduled to deploy on the node.
       Existing pods will continue to run until completed or evacuated (see
       other options for 'oadm manage-node').

[Note] Running diagnostic: ServiceExternalIPs
       Description: Check for existing services with ExternalIPs that are disallowed by master config

[Note] Running diagnostic: AnalyzeLogs
       Description: Check for recent problems in systemd service logs

Info:  Checking journalctl logs for 'atomic-openshift-node' service
Info:  Checking journalctl logs for 'docker' service

[Note] Running diagnostic: MasterConfigCheck
       Description: Check the master config file

WARN:  [DH0005 from diagnostic MasterConfigCheck@openshift/origin/pkg/diagnostics/host/check_master_config.go:52]
       Validation of master config file '/etc/origin/master/master-config.yaml' warned:
       assetConfig.loggingPublicURL: Invalid value: "": required to view aggregated container logs in the console
       assetConfig.metricsPublicURL: Invalid value: "": required to view cluster metrics in the console
       auditConfig.auditFilePath: Required value: audit can now be logged to a separate file

[Note] Running diagnostic: NodeConfigCheck
       Description: Check the node config file

Info:  Found a node config file: /etc/origin/node/node-config.yaml

[Note] Running diagnostic: UnitStatus
       Description: Check status for related systemd units

[Note] Summary of diagnostics execution (version v3.6.5.5):
[Note] Warnings seen: 5
[Note] Errors seen: 0

Based on the results of the diagnostics, actions can be taken to alleviate any issues.

This section focuses on the etcd cluster. It describes the different commands to ensure the cluster is healthy. The internal DNS names of the nodes running etcd must be used.

SSH into the first master node (master1) as before:

$ ssh <user>@<resourcegroup>b.<region>.cloudapp.azure.com
$ ssh <user>@master1
$ sudo -i

Using the output of the command hostname issue the etcdctl command to confirm that the cluster is healthy.

# etcdctl --endpoints https://master1:2379,https://master2:2379,https://master3:2379 --ca-file /etc/etcd/ca.crt --cert-file=/etc/origin/master/master.etcd-client.crt --key-file=/etc/origin/master/master.etcd-client.key cluster-health
member 82c895b7b0de4330 is healthy: got healthy result from https://10.0.0.4:2379
member c8e7ac98bb93fe8c is healthy: got healthy result from https://10.0.0.5:2379
member f7bbfc4285f239ba is healthy: got healthy result from https://10.0.0.6:2379

As explained in Nodes section, node labels are an important part of the OpenShift Container Platform environment. By default of the reference architecture installation, the default node selector is set to role=apps in /etc/origin/master/master-config.yaml on all of the master nodes. This configuration parameter is set during the installation of OpenShift on all masters.

SSH into the first master node (master1) to verify the defaultNodeSelector is defined.

$ ssh <user>@<resourcegroup>b.<region>.cloudapp.azure.com
$ ssh <user>@master1
$ sudo -i
# vi /etc/origin/master/master-config.yaml
... [OUTPUT ABBREVIATED] ...
projectConfig:
  defaultNodeSelector: "role=app"
  projectRequestMessage: ""
  projectRequestTemplate: ""
... [OUTPUT ABBREVIATED] ...

Quotas are set on ephemeral volumes within pods to prohibit a pod from becoming too large and impacting the node. There are three places where sizing restrictions should be set. When persistent volume claims are not set a pod has the ability to grow as large as the underlying filesystem will allow. The required modifications are set by automatically.

OpenShift Volume Quota

At launch time a script creates a XFS partition on the block device, adds an entry in /etc/fstab, and mounts the volume with the option of gquota. If gquota is not set the OpenShift Container Platform node will not be able to start with the perFSGroup parameter defined below. This disk and configuration is done on the master, infrastructure, and application nodes.

SSH into the first infrastructure node (infranode1) to verify the entry exists within /etc/fstab

$ ssh <user>@<resourcegroup>b.<region>.cloudapp.azure.com
$ ssh <user>@infranode1
$ grep "/var/lib/origin/openshift.local.volumes" /etc/fstab
/dev/sdc1 /var/lib/origin/openshift.local.volumes xfs gquota 0 0

OpenShift Emptydir Quota

During installation a value for perFSGroup is set within the node configuration. The perFSGroup setting restricts the ephemeral emptyDir volume from growing larger than 512Mi. This emptyDir quota is done on the master, infrastructure, and application nodes.

SSH into the first infrastructure node (infranode1) to verify /etc/origin/node/node-config.yml matches the information below.

$ ssh <user>@<resourcegroup>b.<region>.cloudapp.azure.com
$ ssh <user>@infranode1
$ sudo grep -B2 perFSGroup /etc/origin/node/node-config.yaml
volumeConfig:
  localQuota:
     perFSGroup: 512Mi

Docker Storage Setup

The /etc/sysconfig/docker-storage-setup file is created at launch time by the bash script on every node. This file tells the Docker service to use a specific volume group for containers. Docker storage setup is performed on all master, infrastructure, and application nodes.

SSH into the first infrastructure node (infranode1) to verify /etc/sysconfig/docker-storage-setup matches the information below.

$ ssh <user>@<resourcegroup>b.<region>.cloudapp.azure.com
$ ssh <user>@infranode1
$ cat /etc/sysconfig/docker-storage-setup
DEVS=/dev/sdd
VG=docker_vol
DATA_SIZE=95%VG
STORAGE_DRIVER=overlay2
CONTAINER_ROOT_LV_NAME=dockerlv
CONTAINER_ROOT_LV_MOUNT_PATH=/var/lib/docker
CONTAINER_ROOT_LV_SIZE=100%FREE

In section Required Channels the specific repositories for a successful OpenShift Container Platform installation were defined. All systems except for the bastion host should have the same repositories configured. To verify subscriptions match those defined in Required Channels perform the following. The repositories below are enabled during the rhsm-repos playbook during the installation. The installation will be unsuccessful if the repositories are missing from the system.

SSH into the first infrastructure node (infranode1) and verify the command output matches the information below.

$ ssh <user>@<resourcegroup>b.<region>.cloudapp.azure.com
$ ssh <user>@infranode1
$ yum repolist
Loaded plugins: langpacks, product-id, search-disabled-repos
repo id                                  repo name                                                     status
rhel-7-fast-datapath-rpms/7Server/x86_64 Red Hat Enterprise Linux Fast Datapath (RHEL 7 Server) (RPMs) 27
rhel-7-server-extras-rpms/x86_64         Red Hat Enterprise Linux 7 Server - Extras (RPMs)             461+4
rhel-7-server-ose-3.6-rpms/x86_64        Red Hat OpenShift Container Platform 3.6 (RPMs)               437+30
rhel-7-server-rpms/7Server/x86_64        Red Hat Enterprise Linux 7 Server (RPMs)                      14.285
repolist: 15.210

This section will cover logging into the OpenShift Container Platform management console via the GUI and the CLI. After logging in via one of these methods applications can then be deployed and managed.

$ oc login https://resourcegroupname.region.cloudapp.azure.com --token=fEAjn7LnZE6v5SOocCSRVmUWGBNIIEKbjD9h-Fv7p09

$ oc new-project test-app
$ oc new-app https://github.com/openshift/cakephp-ex.git --name=php
--> Found image 2997627 (7 days old) in image stream "php" in project "openshift" under tag "5.6" for "php"

    Apache 2.4 with PHP 5.6
    -----------------------
    Platform for building and running PHP 5.6 applications

    Tags: builder, php, php56, rh-php56

    * The source repository appears to match: php
    * A source build using source code from https://github.com/openshift/cakephp-ex.git will be created
      * The resulting image will be pushed to image stream "php:latest"
    * This image will be deployed in deployment config "php"
    * Port 8080/tcp will be load balanced by service "php"
      * Other containers can access this service through the hostname "php"

--> Creating resources with label app=php ...
    imagestream "php" created
    buildconfig "php" created
    deploymentconfig "php" created
    service "php" created
--> Success
    Build scheduled, use 'oc logs -f bc/php' to track its progress.
    Run 'oc status' to view your app.

$ oc expose service php
route "php" exposed

$ oc status
In project test-app on server https://resourcegroupname.region.cloudapp.azure.com

http://test-app.apps.13.93.162.100.nip.io to pod port 8080-tcp (svc/php)
  dc/php deploys istag/php:latest <- bc/php builds https://github.com/openshift/cakephp-ex.git with openshift/php:5.6
    deployment #1 deployed about a minute ago - 1 pod

In this section, reactions to failure are explored. After a successful install and some of the smoke tests noted above have been completed, failure testing is executed.

$ ssh <user>@<resourcegroup>b.<region>.cloudapp.azure.com
$ ssh <user>@master1
$ sudo -i
# etcdctl --endpoints https://master1:2379,https://master2:2379,https://master3:2379 --ca-file /etc/etcd/ca.crt --cert-file=/etc/origin/master/master.etcd-client.crt --key-file=/etc/origin/master/master.etcd-client.key cluster-health
failed to check the health of member 82c895b7b0de4330 on https://10.20.2.251:2379: Get https://10.20.1.251:2379/health: dial tcp 10.20.1.251:2379: i/o timeout
member 82c895b7b0de4330 is unreachable: [https://10.20.1.251:2379] are all unreachable
member c8e7ac98bb93fe8c is healthy: got healthy result from https://10.20.3.74:2379
member f7bbfc4285f239ba is healthy: got healthy result from https://10.20.1.106:2379
cluster is healthy

$ oc get projects
NAME               DISPLAY NAME   STATUS
openshift                         Active
openshift-infra                   Active
ttester                           Active
test-app1                         Active
default                           Active
management-infra                  Active

$ oc project test-app1
Now using project "test-app1" on server "https://resourcegroupname.region.cloudapp.azure.com".

$ oc status
In project test-app1 on server https://resourcegroupname.region.cloudapp.azure.com

http://test-app1.apps.13.93.162.100.nip.io to pod port 8080-tcp (svc/php-prod)
  dc/php-prod deploys istag/php-prod:latest <-
    bc/php-prod builds https://github.com/openshift/cakephp-ex.git with openshift/php:5.6
    deployment #1 deployed 27 minutes ago - 1 pod

$ oc get pods -n default | grep docker-registry
docker-registry-4-9r033    1/1       Running   0          2h

$ oc exec docker-registry-4-9r033 cat /etc/secrets/registry.crt >> /tmp/my-docker-registry-certificate.crt

$ oc get route docker-registry -n default
NAME              HOST/PORT                                      PATH      SERVICES          PORT      TERMINATION   WILDCARD
docker-registry   docker-registry-default.13.64.245.134.nip.io             docker-registry   <all>     passthrough   None

$ sudo mkdir -p /etc/docker/certs.d/docker-registry-default.13.64.245.134.nip.io

$ sudo mv /tmp/my-docker-registry-certificate.crt /etc/docker/certs.d/docker-registry-default.13.64.245.134.nip.io/ca.crt
$ sudo systemctl restart docker

$ oc whoami -t
feAeAgL139uFFF_72bcJlboTv7gi_bo373kf1byaAT8

$ docker pull fedora/apache
$ docker images | grep fedora/apache
docker.io/fedora/apache  latest  c786010769a8  3 months ago  396.4 MB

$ docker tag docker.io/fedora/apache docker-registry-default.13.64.245.134.nip.io/openshift/prodapache

$ docker images | grep openshift/prodapache
docker-registry-default.13.64.245.134.nip.io/openshift/prodapache   latest              c786010769a8        3 months ago        396.4 MB

$ docker login -u $(oc whoami) -e <email> -p $(oc whoami -t) docker-registry-default.13.64.245.134.nip.io
Login Succeeded

$ docker push docker-registry-default.13.64.245.134.nip.io/openshift/prodapache
The push refers to a repository [docker-registry-default.13.64.245.134.nip.io/openshift/prodapache]
3a85ee80fd6c: Pushed
5b0548b012ca: Pushed
a89856341b3d: Pushed
a839f63448f5: Pushed
e4f86288aaf7: Pushed
latest: digest: sha256:e2a15a809ce2fe1a692b2728bd07f58fbf06429a79143b96b5f3e3ba0d1ce6b5 size: 7536

$ oc get pods -o wide -n default
NAME                       READY     STATUS    RESTARTS   AGE       IP           NODE
docker-registry-4-9r033    1/1       Running   0          2h        10.128.6.5   infranode3
registry-console-1-zwzsl   1/1       Running   0          5d        10.131.4.2   infranode2
router-1-09x4g             1/1       Running   0          5d        10.0.2.5     infranode2
router-1-6135c             1/1       Running   0          5d        10.0.2.4     infranode1
router-1-l2562             1/1       Running   0          5d        10.0.2.6     infranode3

$ oc get pods -o wide -n default | grep docker-registry
docker-registry-4-kd40f    1/1       Running   0          1m        10.130.4.3   infranode1

Red Hat OpenShift Container Platform metrics components enable additional features in the Red Hat OpenShift Container Platform web interface. If the environment has been deployed choosing to deploy metrics, there will be a new tab in the pod section named "Metrics" where it shows usage data of CPU, memory and network resources for a period of time:

Using the CLI, the cluster-admin can observe the usage of the pods and nodes using the following commands as well:

$ oc adm top pod --heapster-namespace="openshift-infra" --heapster-scheme="https" --all-namespaces
NAMESPACE         NAME                                   CPU(cores)   MEMORY(bytes)
openshift-infra   hawkular-cassandra-1-h9mrq             161m         1423Mi
logging           logging-fluentd-g5jqw                  8m           92Mi
logging           logging-es-ops-b44n3gav-1-zkl3r        19m          861Mi
... [OUTPUT ABBREVIATED] ...
$ oc adm top node --heapster-namespace="openshift-infra" --heapster-scheme="https"
NAME         CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%
infranode3   372m         9%        4657Mi          33%
master3      68m          1%        1923Mi          13%
node02       43m          1%        1437Mi          5%
... [OUTPUT ABBREVIATED] ...

$ oc new-project autoscaletest
Now using project "autoscaletest" on server "https://myocp.eastus2.cloudapp.azure.com:8443".
... [OUTPUT ABBREVIATED] ...

$ oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git
--> Found Docker image d9c9735 (10 days old) from Docker Hub for "centos/ruby-22-centos7"
... [OUTPUT ABBREVIATED] ...

$ oc patch dc/ruby-ex -p \'{"spec":{"template":{"spec":{"containers":[{"name":"ruby-ex","resources":{"limits":{"cpu":"80m"}}}]}}}}'
"ruby-ex" patched

$ oc get pods
NAME              READY     STATUS      RESTARTS   AGE
ruby-ex-1-210l9   1/1       Running     0          2m
ruby-ex-1-build   0/1       Completed   0          4m

$ oc describe pod ruby-ex-1-210l9
Name:			ruby-ex-1-210l9
... [OUTPUT ABBREVIATED] ...
    Limits:
      cpu:	80m
    Requests:
      cpu:		80m

$ oc autoscale dc/ruby-ex --min 1 --max 10 --cpu-percent=50
deploymentconfig "ruby-ex" autoscaled
$ oc get horizontalpodautoscaler
NAME      REFERENCE                  TARGET    CURRENT   MINPODS   MAXPODS   AGE
ruby-ex   DeploymentConfig/ruby-ex   50%       0%        1         10        53s

$ oc rsh ruby-ex-1-210l9

sh-4.2$ while true; do echo "cpu hog" >> mytempfile; rm -f mytempfile; done

$ oc get events -w
LASTSEEN                        FIRSTSEEN                       COUNT     NAME      KIND                      SUBOBJECT   TYPE      REASON                    SOURCE                         MESSAGE
2017-07-13 13:28:35 +0000 UTC   2017-07-13 13:26:30 +0000 UTC   7         ruby-ex   HorizontalPodAutoscaler               Normal    DesiredReplicasComputed   {horizontal-pod-autoscaler }   Computed the desired num of replicas: 0 (avgCPUutil: 0, current replicas: 1)
2017-07-13 13:29:05 +0000 UTC   2017-07-13 13:29:05 +0000 UTC   1         ruby-ex   HorizontalPodAutoscaler             Normal    DesiredReplicasComputed   {horizontal-pod-autoscaler }   Computed the desired num of replicas: 2 (avgCPUutil: 67, current replicas: 1)
2017-07-13 13:29:05 +0000 UTC   2017-07-13 13:29:05 +0000 UTC   1         ruby-ex   DeploymentConfig             Normal    ReplicationControllerScaled   {deploymentconfig-controller }   Scaled replication controller "ruby-ex-1" from 1 to 2
2017-07-13 13:29:05 +0000 UTC   2017-07-13 13:29:05 +0000 UTC   1         ruby-ex   HorizontalPodAutoscaler             Normal    SuccessfulRescale   {horizontal-pod-autoscaler }   New size: 2; reason: CPU utilization above target
2017-07-13 13:29:05 +0000 UTC   2017-07-13 13:29:05 +0000 UTC   1         ruby-ex-1-zwmxd   Pod                 Normal    Scheduled   {default-scheduler }   Successfully assigned ruby-ex-1-zwmxd to node02

$ oc get pods
NAME              READY     STATUS      RESTARTS   AGE
ruby-ex-1-210l9   1/1       Running     0          8m
ruby-ex-1-build   0/1       Completed   0          9m
ruby-ex-1-zwmxd   1/1       Running     0          58s

$ oc get events -w
LASTSEEN                        FIRSTSEEN                       COUNT     NAME      KIND                      SUBOBJECT   TYPE      REASON                    SOURCE                         MESSAGE
2017-07-13 13:34:05 +0000 UTC   2017-07-13 13:34:05 +0000 UTC   1         ruby-ex   HorizontalPodAutoscaler             Normal    SuccessfulRescale   {horizontal-pod-autoscaler }   New size: 1; reason: All metrics below target
2017-07-13 13:34:05 +0000 UTC   2017-07-13 13:34:05 +0000 UTC   1         ruby-ex   DeploymentConfig             Normal    ReplicationControllerScaled   {deploymentconfig-controller }   Scaled replication controller "ruby-ex-1" from 2 to 1
2017-07-13 13:34:05 +0000 UTC   2017-07-13 13:34:05 +0000 UTC   1         ruby-ex-1   ReplicationController             Normal    SuccessfulDelete   {replication-controller }   Deleted pod: ruby-ex-1-zwmxd

Red Hat OpenShift Container Platform aggregated logging components enable additional features in the Red Hat OpenShift Container Platform web interface. If the environment has been deployed choosing to deploy logging, there will be a new link in the pod logs section named "View Archive" that will redirect to the Kibana web interface for the user to see the pods logs, create queries, filters, etc.

In case the "opslogging" cluster has been deployed, there will be a route "kibana-ops" in the "logging" project where cluster-admin users can browse infrastructure logs.

Container storage is defined by the concept of persistent volumes (pv) which are OpenShift Container Platform objects that allow for storage to be defined and then used by pods for data persistence. Requesting of persistent volumes is done by using a persistent volume claim (pvc) object. This claim, when successfully fulfilled by the system will also mount the persistent storage to a specific directory within a pod or multiple pods. This directory is referred to as the mountPath and facilitated using a concept known as bind-mount.

Persistent volumes can be preprovisioned by the OpenShift Container Platform administrator by creating them in the underlying infrastructure and in OpenShift Container Platform manually, or the administrator can configure OpenShift Container Platform to create automatically the proper persistent volumes when users request them using the dynamic provisioning and storage classes capabilities of OpenShift Container Platform.

The StorageClass resource object describes and classifies different types of storage that can be requested, as well as provides a means for passing parameters to the backend for dynamically provisioned storage on demand. StorageClass objects can also serve as a management mechanism for controlling different levels of storage and access to the storage. Cluster Administrators (cluster-admin) or Storage Administrators (storage-admin) define and create the StorageClass objects that users can use without needing any intimate knowledge about the underlying storage volume sources. Because of this the naming of the storage class defined in the StorageClass object should be useful in understanding the type of storage it maps to (ie., HDD vs SDD or Premium_LRS vs Standard_LRS).

Cloud provider specific storage is storage that is provided from Microsoft Azure. This type of storage is presented as an Data disk VHD and can be mounted by one pod at a time. It is needed to configure OpenShift Container Platform with Microsoft Azure settings like the resourceGroup or subscriptionID in the /etc/azure/azure.conf file on masters and nodes as well as in the OpenShift Container Platform masters and nodes configuration file to be able to use VHD as persistent storage for pods. The settings needed are automatically configured as part of the installation process using the code provided in the openshift-ansible-contrib git repository.

Cloud provider storage can be created manually and assigned as a persistent volume or a persistent volume can be created dynamically using a StorageClass object. Note that VHD storage can only use the access mode of Read-Write-Once (RWO).

The VHDs used in Microsoft Azure are .vhd files stored as page blobs in a standard or premium storage account in Microsoft Azure where standard delivers cost-effective storage and premium delivers high-performance, low-latency storage.

$ azure storage account create --sku-name <sku> --kind "Storage" -g <resourcegroup> -l <region> <storage account name>

$ azure storage account create --sku-name "LRS" --kind "Storage" -g sysdeseng sapv3sysdeseng -l "westus"
info:    Executing command storage account create
+ Checking availability of the storage account name
+ Creating storage account
info:    storage account create command OK

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: mystorageclass
provisioner: kubernetes.io/azure-disk
parameters:
  storageAccount: sapv3sysdeseng

$ oc create -f my-storage-class.yaml

$ vi db-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: db
 annotations:
   volume.beta.kubernetes.io/storage-class: mystorageclass
spec:
 accessModes:
  - ReadWriteOnce
 resources:
   requests:
     storage: 10Gi

$ oc create -f db-claim.yaml
persistentvolumeclaim "db" created
$ oc get pvc db
NAME      STATUS    VOLUME                                     CAPACITY   ACCESSMODES   AGE
db        Bound     pvc-be63668e-451e-11e7-b30b-000d3a36dea3   10Gi       RWO           1m

$ oc describe pv pvc-be63668e-451e-11e7-b30b-000d3a36dea3
Name:		pvc-be63668e-451e-11e7-b30b-000d3a36dea3
Labels:		<none>
StorageClass:	mystorageclass
Status:		Bound
Claim:		testdev/db
Reclaim Policy:	Delete
Access Modes:	RWO
Capacity:	10Gi
Message:
Source:
    Type:		AzureDisk (an Azure Data Disk mount on the host and bind mount to the pod)
    DiskName:		kubernetes-dynamic-pvc-be63668e-451e-11e7-b30b-000d3a36dea3.vhd
    DiskURI:		https://sapv3sysdeseng.blob.core.windows.net/vhds/kubernetes-dynamic-pvc-be63668e-451e-11e7-b30b-000d3a36dea3.vhd
    FSType:		ext4
    CachingMode:	None
    ReadOnly:		false
No events.

$ oc delete pvc db
persistentvolumeclaim "db" deleted
$ oc get pvc db
No resources found.
Error from server: persistentvolumeclaims "db" not found

Verify the quantity and type of the nodes in the cluster by using the oc get nodes command. The output below is an example of a complete Red Hat OpenShift Container Platform environment after the deployment of the reference architecture environment.

$ oc get nodes
NAME         STATUS                     AGE
infranode1   Ready                      3m
infranode2   Ready                      3m
infranode3   Ready                      3m
master1      Ready,SchedulingDisabled   3m
master2      Ready,SchedulingDisabled   3m
master3      Ready,SchedulingDisabled   3m
node01       Ready                      3m
node02       Ready                      3m
node03       Ready                      3m

The script should be executed as the regular user created as part of the Red Hat OpenShift Container Platform reference architecture deployment on the bastion host.

The bash script add_host.sh adds new hosts to the Red Hat OpenShift Container Platform cluster where it accepts a few parameters to customize the new host that is going to be created in Microsoft Azure as part of the process. The script creates the required Microsoft Azure components such as nic, vm, nsg, attaches the new host to the load balancer if needed, deploys the vm as part of the same availabilityset, etc., run the prerrequisites playbooks to prepare the host, modifies the ansible inventory to fit the requirements and then runs the proper scale up playbook provided by the atomic-openshift-utils package.

To add an Application Node with the default values, run the add_host.sh script following the example below. Once the instance is launched, the installation of Red Hat OpenShift Container Platform will automatically begin.

$ ./add_host.sh

If some parameters need to be customized, use the proper flags. The example below adds a new Application Node with a different VM size, different user and ssh public key:

$ ./add_host.sh -s Standard_DS4_v2 -u user123 -p /my/other/ssh-id.pub

The process for adding an Infrastructure Node is nearly identical to adding an Application Node. The only difference in adding an Infrastructure node is the type flag need to be set to "infranode". Follow the example steps below to add a new Infrastructure Node using the default values:

$ ./add_host.sh -t infranode

If some parameters need to be customized, use the proper flags. The example below adds a new Infrastructure Node with different disk sizes:

$ ./add_host.sh -t infranode -d 20 -d 200

The process for adding a Master host is nearly identical to adding an Application Node. The only difference in adding a Master host is the type flag need to be set to "master". Follow the example steps below to add a new Master host using the default values:

$ ./add_host.sh -t master

If some parameters need to be customized, use the proper flags. The example below adds a new Master Host with different disk sizes and different user:

$ ./add_host.sh -t master -d 50 -d 20 -u adminxyz

To verify a newly provisioned host has been added to the existing environment, use the oc get nodes command. In this example, 2 new Infrastructure nodes, 2 new Master hosts and 2 new Application Nodes have been added using the add_host.sh script by executing it a few times.

$ oc get nodes
NAME         STATUS                     AGE
infranode1   Ready                      5h
infranode2   Ready                      5h
infranode3   Ready                      5h
infranode4 Ready 1h
infranode5 Ready 4m
master1      Ready,SchedulingDisabled   5h
master2      Ready,SchedulingDisabled   5h
master3      Ready,SchedulingDisabled   5h
master4 Ready,SchedulingDisabled 2h
master5 Ready,SchedulingDisabled 1h
node01       Ready                      5h
node02       Ready                      5h
node03       Ready                      5h
node04 Ready 3h
node05 Ready 2h

The following procedure creates a new project and forces the pods of that project to run on the new host. This procedure validates the host is properly configured to run Red Hat OpenShift Container Platform pods:

Create a new project to test:

$ oc new-project scaleuptest
Now using project "scaleuptest" on server "https://myocpdeployment.eastus2.cloudapp.azure.com:8443".
... [OUTPUT ABBREVIATED] ...

Patch the node-selector to only run pods on the new node:

$ oc patch namespace scaleuptest -p "{\"metadata\":{\"annotations\":{\"openshift.io/node-selector\":\"kubernetes.io/hostname=node04\"}}}"
"scaleuptest" patched

Deploy an example app:

$ oc new-app openshift/hello-openshift
--> Found Docker image 8146af6 (About an hour old) from Docker Hub for "openshift/hello-openshift"
... [OUTPUT ABBREVIATED] ...

Scale the number of pods to ensure they are running on the same host:

$ oc scale dc/hello-openshift --replicas=8
deploymentconfig "hello-openshift" scaled

Observe where the pods run:

$ oc get pods -o wide
hello-openshift-1-1ffl6   1/1       Running   0          3m        10.128.4.10   node04
hello-openshift-1-1kgpf   1/1       Running   0          3m        10.128.4.3    node04
hello-openshift-1-4lk85   1/1       Running   0          3m        10.128.4.4    node04
hello-openshift-1-4pfkk   1/1       Running   0          3m        10.128.4.7    node04
hello-openshift-1-56pqg   1/1       Running   0          3m        10.128.4.6    node04
hello-openshift-1-r3sjz   1/1       Running   0          3m        10.128.4.8    node04
hello-openshift-1-t0fmm   1/1       Running   0          3m        10.128.4.5    node04
hello-openshift-1-v659g   1/1       Running   0          3m        10.128.4.9    node04

Clean the environment:

$ oc delete project scaleuptest

In case the checks are mandatory before adding the host to the cluster, the labels can be set to avoid the default node-selector, run the checks then relabel the node:

... [OUTPUT ABBREVIATED] ...
[new_nodes]
node04.example.com openshift_node_labels="{\'role': \'test',\'test':\'true'}"

Perform the scale up procedure, run the required tests, then relabel the node:

$ oc label node node04 "role=app" "zone=X" --overwrite
node "node04" labeled
$ oc label node node04 test-
node "node04" labeled

This wealth of features makes Red Hat Single Sign-On an ideal identity and authentication provider as it includes the ability to authenticate developers combined with ability to authenticate and control applications running on Red Hat OpenShift Container Platform.

The bastion.sh ( or allinone.sh for single node deployment ) script creates an ansible playbook setup-sso.yaml, which is executed automatically after the deployment and creates all the required elements in the Red Hat OpenShift Container Platform cluster:

In the login screen of the Red Hat OpenShift Container Platform web interface there is now a selection screen, allowing the user to select either htpasswd based authentication or sso based authentication.

  - challenge: true
    login: true
    mappingMethod: claim
    name: htpasswd_auth
    provider:
      apiVersion: v1
      file: /etc/origin/master/htpasswd
      kind: HTPasswdPasswordIdentityProvider

When user selects sso as the authentication method, a redirection to the sso cloud login page happens. The login screen is provided by the sso pod running in the Red Hat OpenShift Container Platform cluster.

During deployment a user is created in both htpasswd and sso configuration to allow the administration of the cluster. Both accounts use the the username and password that was supplied to the ARM template. In addition, an admin user is created in the sso using the same password supplied during the running of the reference architecture.

After completion of the reference architecture, a route in the SSO project is created that contains the url to access the web interface of the Red Hat Single Sign-On pod.

In this example, the url is https://login.13.76.83.116.nip.io. The URL to access the web interface of Red Hat Single Sign-On is based on the public IP of the load balancer, created automatically during the execution of the reference architecture. Accessing the route URL shows the login screen of the Red Hat Single Sign-On application where the user name is "admin" and the password is the same as the supplied as a parameter (adminPassword).

After logging into SSO Application, the administrator web interface is shown as:

The full documenation of SSO can be found at: RedHat SSO 7.1 Server Administration Guide.

The OpenShift Container Platform installation can be controlled from the bastion host. This is a separate virtual machine that allows access to all VM’s in the same resource group that defines the OpenShift Container Platform installation on Microsoft Azure.

As an example, assuming the resource group was named during creation to ocpxenon1000, with a username of ocpadmin on the westus region:

$ ssh ocpadmin@ocpxenon1000b.westus.cloudapp.azure.com
Last login: Sat Jan 21 04:32:47 2017 from 103.252.201.32
[ocpadmin@bastion ~]$

The automation collects logs for various stages of the installation. All the logs are stored on the bastion host. Assuming ocpadmin has been chosen as the admin user when creating the install, there are some useful logs in the home directory of the user (/home/ocpadmin):

The inventory for ansible is automatically generated by the bastion.sh script at the first boot of the bastion host, stored in the bastion itself at the default location (/etc/ansible/hosts) and can be used to run update scripts. In order to run updates, or to diagnose failures, it is necessary to ssh to the bastion host on Microsoft Azure.

$ sudo cat /etc/ansible/hosts
[OSEv3:children]
masters
etcd
nodes
misc

[OSEv3:vars]
azure_resource_group=ocpxenon1000
rhn_pool_id=8a85f98156724eaa0156728452003452
openshift_install_examples=true
deployment_type=openshift-enterprise
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]

# default selectors for router and registry services
openshift_router_selector='region=infra'
openshift_registry_selector='region=infra'

ansible_become=yes
ansible_ssh_user=ocpadmin
remote_user=ocpadmin

openshift_master_default_subdomain=52.163.224.147.xip.io
openshift_use_dnsmasq=False
openshift_public_hostname=ocpxenon1000.westus.cloudapp.azure.com

openshift_master_cluster_method=native
openshift_master_cluster_hostname=ocpxenon1000.westus.cloudapp.azure.com
openshift_master_cluster_public_hostname=ocpxenon1000.westus.cloudapp.azure.com

# Enable cockpit
osm_use_cockpit=true

# Set cockpit plugins
osm_cockpit_plugins=['cockpit-kubernetes']

# default storage plugin dependencies to install, by default the ceph and
# glusterfs plugin dependencies will be installed, if available.
osn_storage_plugin_deps=['Azure VHD']

[masters]
master1 openshift_hostname=master1 openshift_node_labels="{'role': 'master'}"
master2 openshift_hostname=master2 openshift_node_labels="{'role': 'master'}"
master3 openshift_hostname=master3 openshift_node_labels="{'role': 'master'}"

[etcd]
master1
master2
master3

[nodes]
master1 openshift_node_labels="{'region':'master','zone':'default'}" openshift_schedulable=false
master2 openshift_node_labels="{'region':'master','zone':'default'}" openshift_schedulable=false
master3 openshift_node_labels="{'region':'master','zone':'default'}" openshift_schedulable=false
node[01:03] openshift_node_labels="{'role': 'app', 'zone': 'default'}"
infranode1 openshift_hostname=infranode1 openshift_node_labels="{'role': 'infra', 'zone': 'default'}"
infranode2 openshift_hostname=infranode2 openshift_node_labels="{'role': 'infra', 'zone': 'default'}"
infranode3 openshift_hostname=infranode3 openshift_node_labels="{'role': 'infra', 'zone': 'default'}"

The uninstall playbook removes OpenShift Container Platform related packages, etcd, and removes any certificates that were created during the failed install. In case you need to do it, run the following from the bastion host:

$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/adhoc/uninstall.yml

After the playbook, the administrator should unsubscribe each host, to return the subscription back into the available pool, and then delete the resource group within Microsoft Azure portal, which will delete all resources.

The openshift-install.sh script, located in the bastion host at /home/user/openshift-install.sh, can be used to automatically install OpenShift Container Platform. The script can be re-run to diagnose problems.

$ ./openshift-install.sh

The bastion.sh script can optionally notify the user via email during the installation about the steps that has been done. It creates a /root/setup_ssmtp.sh script with the username and password provided in the ARM template that will configure an ssmtp MTA service, and if the GMail account exists, it will notify the user periodically on the steps finished.