Chapter 3. Provisioning the Infrastructure
This chapter focuses on Phase 1 of the process. The prerequisites defined below are required for a successful deployment of infrastructure and the installation of OpenShift Container Platform on Microsoft Azure.
3.1. Provisioning Prerequisites
The script and playbooks provided within the git repository deploys infrastructure, installs and configures OpenShift Container Platform, and performs post installation tasks such as scaling the router and registry. The playbooks create specific roles, policies, and users required for cloud provider configuration in OpenShift Container Platform, as well as storage accounts on Microsoft Azure. In order to deploy OpenShift Container Platform on Microsoft Azure, an ARM template must be created. The ARM template takes the basic information, admin user name, password, Red Hat Subscription Manager username and password, SSH public and private keys, as well as offering the ability to size the virtual machines.
3.1.1. Authentication Prerequisites
The template creates an OpenShift Container Platform user using the supplied values adminUsername and adminPassword in the ARM template parameters when creating the infrastructure and it will be granted the cluster-admin role for the OpenShift Container Platform environment.
3.1.2. SSH Prerequisites
3.1.2.1. SSH Configuration
The SSH service will be configured during the deployment to allow connections using a public key. To make this work, the same public and private key that were created before will be used for the admin user in the instances. Before beginning the deployment of the Microsoft Azure infrastructure and the deployment of OpenShift Container Platform, the SSHprivate key must be converted into a base64 string. Otherwise, the Microsoft Azure infrastructure cannot be configured manually or automatically.
The reason to convert the private key to a base64 string is to avoid possible issues with multiple lines and carriage return characters
This task is performed locally, on the machine that will run the ARM template. In this example, the private key is converted to base64 encoding, and placed on the clipboard.
- For RHEL/CentOS:
$ cat ~/.ssh/id_rsa | base64 | tr -d '\n' | xclip -selection clipboard- For OSX:
$ cat ~/.ssh/id_rsa | base64 | tr -d '\n' | pbcopy- For Windows use a Base 64 Encoder app or website to perform the conversion
3.1.3. Red Hat Subscription Prerequisites
A Red Hat account is required to use this template. This account must have appropriate subscriptions available in the account in order to use the template. Either a username and password may be supplied, or an organization ID and activation key.
To be able to access the RPM packages the Pool ID for the subscriptions is required. The next section will show how to locate the Pool ID.
3.1.3.1. Red Hat Subscription Pool ID
In order to assign a subscription, a Pool ID is required. Using a system that currently has a valid subscription to Red Hat products, by using subscription-manager the Pool ID can be acquired. Perform the following command, and take note of the Pool ID.
# subscription-manager list --available
+-------------------------------------------+
Available Subscriptions
+-------------------------------------------+
Subscription Name: Red Hat OpenShift Container Platform, Premium (1-2 Sockets)
Provides: Red Hat OpenShift Enterprise Application Node
Red Hat OpenShift Container Platform
Red Hat OpenShift Enterprise Client Tools
Red Hat Enterprise Linux Server
Red Hat Enterprise Linux Atomic Host
Red Hat OpenShift Enterprise Infrastructure
Red Hat CloudForms
SKU: MCT2862
Contract: 38154141
Pool ID: 1b152951269116311123bc522341e1
Provides Management: No
Available: 64
Suggested: 1
Service Level: Premium
Service Type: L1-L3
Subscription Type: Stackable
Ends: 25/08/2017
System Type: Virtual
The Pool ID is also available in the Subscriptions section of the Red Hat Customer Portal, by selecting the appropriate subscription that will open a detailed view of the subscription, including the Pool ID
3.1.4. Organization ID and Activation Key
Instead using username/password combination, an activation key and organization ID can be used. The activation key is a simple string, chosen by the end-user that contains the subscriptions the user attaches to it to avoid using passwords in the registration procedure. The organization ID is the Red Hat ID for the customer organizations and it is required when using activation keys to identify the activation keys within the organization.
Perform the following steps to obtain the organization ID and activation key.
3.1.4.1. Generating Activation Key
- Go to the Red Hat Customer Portal located in http://access.redhat.com and login with your Red Hat username and password. Select [Subscriptions] section in the top left corner then the tab "Activation keys"
- Create your key selecting the "New Activation Key" button
Once in there give your key a name, set service level to standard or premium, and add the subscriptions containing OpenShift Container Platform.
3.1.4.2. Getting the Organization ID
In order to find the organization ID, a RHEL server is required. Perform the following on a existing RHEL server.
$ subscription-manager identity
system identity: 8c85ab6d-495c-451e-b38e-5f09bc3342a0
name: bastion
org name: 6616399
org ID: 6616399
Find the value labeled org ID: and save this somewhere for use during the deployment.
3.1.4.3. Using the Organization ID and Activation Key
When running the ARM template, use the organization ID as the RHN User Name, and use the activation key as the RHN Password.
3.1.5. Azure Active Directory Credentials
It is required to create a Microsoft Azure service principal in order to be able to deploy the infrastructure using the ARM template. The service principal object defines the policy and permissions for an application’s use in a specific tenant, providing the basis for a security principal to represent the application at run-time.
For more information about the service principal objects in Azure Active Directory see Use Azure CLI to create a service principal to access resources
In order to create the Azure Active Directory (AAD) client id, and password, the Node.js Azure CLI package is required.
Follow this instructions to create it in a RHEL/CentOS/Fedora system:
Install Node.js
$ sudo yum -y install npmInstall the
Azure CLINode.js package:$ sudo npm install -g azure-cliLogin to Microsoft Azure:
$ azure loginCreate a service principal:
$ azure ad sp create -n <service_principal_name> -p <password>The following is an example output:
$ azure ad sp create -n openshiftcloudprovider -p Pass@word1 info: Executing command ad sp create + Creating application openshift demo cloud provider + Creating service principal for application 198c4803-1236-4c3f-ad90-46e5f3b4cd2a data: Object Id: 00419334-174b-41e8-9b83-9b5011d8d352 data: Display Name: openshiftcloudprovider data: Service Principal Names: data: 198c4803-1236-4c3f-ad90-46e5f3b4cd2a data: http://myhomepage info: ad sp create command OKSave the
Object Idand theService Principal Names GUIDvalues from the command output.-
The
Object Idwill be used to create the role assignment. -
The
Service Principal Names GUIDwill be used as theaadClientIdparameter value (Application ID/Client ID) in the template. -
The password entered as part of the CLI command will be the
aadClientSecretparamter value in the template.
-
The
Show the Microsoft Azure account data:
$ azure account showThe following is an example output:
$ azure account show info: Executing command account show data: Name : Microsoft Azure Sponsorship data: ID : 2581564b-56b4-4512-a140-012d49dfc02c data: State : Enabled data: Tenant ID : 77ece336-c110-470d-a446-757a69cb9485 data: Is Default : true data: Environment : AzureCloud data: Has Certificate : Yes data: Has Access Token : Yes data: User name : ssysone@something.com data: info: account show command OKSave the command output
IDvalue that will be used for the provisioning.Grant the service principal the access level of
contributorto allow OpenShift Container Platform to create/delete resources using theObject IDandIDparameters from the previous steps$ azure role assignment create --objectId <objectID> -o contributor -c /subscriptions/<id>/The following is an example output:
# azure role assignment create --objectId 00419334-174b-41e8-9b83-9b5011d8d352 -o contributor -c /subscriptions/2581564b-56b4-4512-a140-012d49dfc02c/ info: Executing command role assignment create + Finding role with specified name /data: RoleAssignmentId : /subscriptions/2586c64b-38b4-4527-a140-012d49dfc02c/providers/Microsoft.Authorization/roleAssignments/490c9dd5-0bfa-4b4c-bbc0-aa9af130dd06 data: RoleDefinitionName : Contributor data: RoleDefinitionId : b24988ac-6180-42a0-ab88-20f7382dd24c data: Scope : /subscriptions/2586c64b-38b4-4527-a140-012d49dfc02c data: Display Name : openshiftcloudprovider data: SignInName : undefined data: ObjectId : 00419334-174b-41e8-9b83-9b5011d8d352 data: ObjectType : ServicePrincipal data: + info: role assignment create command OK
3.2. Introduction to the Microsoft Azure Template
Azure Resource Manager templates consist of json files that describes the objects that will be deployed in Microsoft Azure. The main template file for this reference architecture is located in the reference-architecture/azure-ansible/azuredeploy.json file in the git repository. This file is the main ARM template that launches all the other templates under azure-ansible. There are four types of virtual machines created by the template (bastion, master node, infrastructure node and application node) and for each of these types there is a additional json file that defines each VM type.
| Virtual Machine type | Template file |
|---|---|
| Bastion |
|
| Master |
|
| Infrastructure node |
|
| Application node |
|
The ARM template for each type, automatically starts a bash shell script that does part of the initial setup. The main shell script is the reference-architecture/azure-ansible/bastion.sh that handles the generation of the ansible host inventory, as well as the setup and running of ansible across all the hosts. The bastion host also provides isolation of all the hosts in the resource group from the public internet for the purpose of SSH access.
3.3. Alternative Single VM Microsoft Azure Template
In addition to the production template of azuredeploy.json, a single virtual machine version is also available. This template is located at: reference-architecture/azure-ansible/allinone.json This provides for early prototypes and tests of applications in a Red Hat OpenShift Container Platform environment. Note that the single VM does not support the high-availability and load-balancing features of the full azuredeploy.json template. The single virtual machine template only allows you to choose the vm size, and uses a single public ip for console and applications. The Number of Nodes and Wildcard Zone are removed.
3.4. Parameters Required
In order to provision the OpenShift Container Platform environment using the ARM template, the following information is required:
- A Microsoft Azure subscription, with appropriate core and VM quota limits.
-
Resource Group - Used as the name of the OpenShift Container Platform Cluster - All the assets of a single cluster use the
Azure Resource Groupto organize and group the assets. This name needs to be unique for each cluster perAzure Region(Location). As some resources will be created using the resource group name, it must be 3-18 characters in length and use numbers and lower-case letters only. Admin username and password - This will be the admin user, used for multiple purposes:
-
As the
SSHuser to be able to connect to the bastion host, and administer the cluster. - As an OpenShift Container Platform administrative user, able to create and control OpenShift Container Platform from the command line, or the user interface.
-
As the
-
SSH Key Data - This is the public key (
~/.ssh/id_rsa.pub), generated for the user that will be used toSSHaccess to all the VMs. During the creation and installation of OpenShift Container Platform virtual machines, the key will automatically be added to each host. This assures proper security and access. This key must be backed up, as its the only principal way to access the cluster for administration. -
SSH Private Data - This is the private key
~/.ssh/id_rsafile contents that has been base64 encoded. During the creation and installation of OpenShift Container Platform virtual machines, the key will automatically be added to each host. This data should be backed up. -
Wildcard Zone - Subdomain for applications in the OpenShift Container Platform cluster (required by the load balancer, but nip.io will be used). It is just the subdomain, not the full FQDN. Example wildcardzone parameter will be "refarchapps" and the FQDN will be created as
refarchapps.<region>.cloudapp.azure.com - Number of Nodes - The template supports the creation of 3 to 30 nodes during greenfield creation of a cluster. Note that the quota of the Microsoft Azure account must support the number chosen.
-
Image - The template supports
RHEL(Red Hat Enterprise Linux) 7.3 or later. The image will be upgraded during the installation process to the latest release. -
Master VM Size (default:
Standard_DS4_v2) - The default value gives 8 CPU Cores, 28 Gigabytes of memory, with 56 GB of premium storage local disk. This is used for OpenShift Container Platform master nodes, as well as the bastion host. -
Infranode VM Size (default:
Standard_DS4_v2) - The default value gives 8 CPU Cores, 28 Gigabytes of memory, with 56 GB of premium storage local disk. Infrastructure nodes run the OpenShift Container Platform routers and the OpenShift Container Platform registry pods. As the infrastructure nodes provide the ingress for all applications, its recommended that DS2 be the smallest node used for the infrastructure nodes. -
Node VM Size (default:
Standard_DS4_v2) - The default value gives 8 CPU Cores, 28 Gigabytes of memory, with 56 GB of premium storage local disk. Application nodes is where the application containers run. -
RHN Username - This should be the username used for the Red Hat Subscription Account that has OpenShift Container Platform entitlements, or the
Organization IDif using activation keys. -
RHN Password - This should be the password for the Red Hat Subscription Account, or the
activation keyif using activation keys. - Subscription Pool ID - This is a number sequence that uniquely identifies the subscriptions that are to be used for the OpenShift Container Platform installation.
-
AAD Client Id - This gives OpenShift Container Platform the
Active Directory ID, needed to be able to create, move and delete persistent volumes. -
AAD Client Secret - The
Active Directory Passwordto match the AAD Client ID. Required to create persistent volumes. - OpenShiftSDN - The SDN plugin to be used in the environment (ovs-multitenant by default)
- Metrics - Deploy Red Hat OpenShift Container Platform metrics components (true by default)
- Logging - Deploy Red Hat OpenShift Container Platform aggregated logging components (true by default)
- OPS Logging - Deploy Red Hat OpenShift Container Platform ops aggregated logging components (false by default)
3.5. Provision OpenShift Container Platform environment
There are two ways to provision the OpenShift Container Platform environment. Using a Web Interface, by filling out a form generated by the template for the needed parameters. And the alternate way, is using a ansible playbook to deploy the cluster. The ansible method is ideal when you wish to deploy clusters in a repeatable way, or when you wish to have more than one cluster.
3.5.1. Provisioning ARM Template by using the Web Interface
With the above information ready, go to https://github.com/openshift/openshift-ansible-contrib/tree/master/reference-architecture/azure-ansible and click the [Deploy To Azure] button near the bottom of the page. This will then show the form, to allow the deployment to be started.
Figure 3.1. ARM Template

3.5.2. Provisioning ARM Template by using Ansible
The ARM templates may be deployed via Ansible playbook when you wish to have a repeatable method of creating a cluster, or you wish to create multiple clusters.
In the reference scripts, a additional directory is provided, ansibledeployocp. This provides example playbooks to directly create clusters using Ansible in a Linux environment.
First, install git and ansible.
$ sudo yum -y install ansible gitThen clone the openshift-ansible-contrib repository to a RHEL based host.
$ git clone https://github.com/openshift/openshift-ansible-contrib $ cd openshift-ansible-contrib/reference-architecture/azure-ansible/ansibledeployocp/
Next, install the dependencies using the prepare playbook:
$ ansible-playbook playbooks/prepare.yml
Microsoft Azure credentials needs to be stored in a file at ~/.azure/credentials with the following format (do not use quotes or double quotes):
[default] subscription_id=00000000-0000-0000-0000-000000000000 tenant=11111111-1111-1111-1111-111111111111 client_id=33333333-3333-3333-3333-333333333 secret=ServicePrincipalPassword
Where subscription_id and tenant parameters can be obtained from the Microsoft Azure cli:
$ sudo yum install -y nodejs $ sudo npm install -g azure-cli $ azure login $ azure account show info: Executing command account show data: Name : Acme Inc. data: ID : 00000000-0000-0000-0000-000000000000 data: State : Enabled data: Tenant ID : 11111111-1111-1111-1111-111111111111 data: Is Default : true data: Environment : AzureCloud data: Has Certificate : Yes data: Has Access Token : Yes data: User name : youremail@yourcompany.com data: info: account show command OK
The client_id is the "Service Principal Name" parameter when you create the serviceprincipal:
$ azure ad sp create -n azureansible -p ServicePrincipalPassword
info: Executing command ad sp create
+ Creating application ansiblelab
+ Creating service principal for application 33333333-3333-3333-3333-333333333
data: Object Id: 44444444-4444-4444-4444-444444444444
data: Display Name: azureansible
data: Service Principal Names:
data: 33333333-3333-3333-3333-333333333
data: http://azureansible
info: ad sp create command OK
The secret is the serviceprincipal password.
Ansible Parameters required
The ansible playbook needs some parameters to be specified. There is a vars.yaml example file included in this repository that should be customized with your environment data.
PARAMETERS
$ cp vars.yaml.example vars.yaml $ vim vars.yaml
-
sshkeydataid_rsa.pub content -
sshprivatedataid_rsa content in base64 without \n characters (cat ~/.ssh/id_rsa | base64 | tr -d '\n') -
adminusernameUser that will be created to login via ssh and as Red Hat OpenShift Container Platform cluster-admin -
adminpasswordPassword for the user created (in plain text) -
rhsmusernamepasswordoractivationkeyThis should be "usernamepassword" or "activationkey". If "usernamepassword", then the username and password should be specified If "activationkey", then the activation key and organization id should be specified -
rhnusernameThe RHN username where the instances will be registered or "activationkey" if activation key method has been chosen -
rhnpasswordThe RHN password where the instances will be registered in plain text -
rhnpassword"organizationid" if activation key method has been chosen else password. -
subscriptionpoolidThe subscription pool id the instances will use -
resourcegroupnameThe Microsoft Azure resource name that will be created -
aadclientidActive Directory ID needed to be able to create, move and delete persistent volumes -
aadclientsecretThe Active Directory Password to match the AAD Client ID -
wildcardzoneSubdomain for applications in the OpenShift cluster (required by the load balancer, but nip.io will be used). It is just the subdomain, not the full FQDN.
Optional (default values are set in playbooks/roles/azure-deploy/default/main.yaml)
-
templatelink- The ARM template that will be deployed -
numberofnodes- From 3 to 30 nodes -
image- The operating system image that will be used to create the instances -
mastervmsize- Master nodes VM size -
infranodesize- Infrastructure nodes VM size -
nodevmsize- Application nodes VM size -
location- westus by default -
openshiftsdn- "redhat/openshift-ovs-multitenant" by default -
metrics- Deploy Red Hat OpenShift Container Platform metrics components (true by default) -
logging- Deploy Red Hat OpenShift Container Platform aggregated logging components (true by default) -
opslogging- Deploy Red Hat OpenShift Container Platform ops aggregated logging components (false by default)
Running the deploy
$ ansible-playbook -e @vars.yaml playbooks/deploy.yaml PLAY [localhost] TASK [Destroy Azure Deploy] changed: [localhost] TASK [Destroy Azure Deploy] ok: [localhost] TASK [Create Azure Deploy] changed: [localhost] PLAY RECAP ** localhost : ok=3 changed=2 unreachable=0 failed=0
3.6. Post Deployment
Once the playbooks have successfully completed the next steps will be to perform the steps defined in Chapter 4, Operational Management. In the event that OpenShift Container Platform failed to install, follow the steps in Appendix C: Chapter 8, Installation Failure to restart the installation of OpenShift Container Platform.
3.7. Post Provisioning Results
At this point the infrastructure and Red Hat OpenShift Container Platform have been deployed. Log into the Azure web console and check the resources. In the Azure web console, check for the following resources:
- 3 master nodes
- 3 infrastructure nodes
- 3 or more application nodes
- 1 unique virtual network
- 3 public IPs
- 10 network interfaces
- 4 network security groups
- 5 storage accounts
- 2 load balancer profiles
- 2 load balancer DNS entries
- 3 routers
- 3 registries
After the Azure Resource Manager template is submitted, and the ARM deployment succeeds, the ansible install is started automatically.
A wildcard DNS entry must be created if a custom domain is required, by default the nip.io service is be used as explained in the Microsoft Azure DNS section.
When installing using this method the browser certificate must be accepted three times due to the number of masters in the cluster. Failure to accept the certificate can cause disconnect issues and the appearance of network failures.


Where did the comment section go?
Red Hat's documentation publication system recently went through an upgrade to enable speedier, more mobile-friendly content. We decided to re-evaluate our commenting platform to ensure that it meets your expectations and serves as an optimal feedback mechanism. During this redesign, we invite your input on providing feedback on Red Hat documentation via the discussion platform.