Chapter 3. Provisioning the Infrastructure

This chapter focuses on Phase 1 of the process. The prerequisites defined below are required for a successful deployment of infrastructure and the installation of OpenShift Container Platform on Microsoft Azure.

3.1. Provisioning Prerequisites

The script and playbooks provided within the git repository deploys infrastructure, installs and configures OpenShift Container Platform, and performs post installation tasks such as scaling the router and registry. The playbooks create specific roles, policies, and users required for cloud provider configuration in OpenShift Container Platform, as well as storage accounts on Microsoft Azure. In order to deploy OpenShift Container Platform on Microsoft Azure, an ARM template must be created. The ARM template takes the basic information, admin user name, password, Red Hat Subscription Manager username and password, SSH public and private keys, as well as offering the ability to size the virtual machines.

3.1.1. Authentication Prerequisites

The template creates an OpenShift Container Platform user using the supplied values adminUsername and adminPassword in the ARM template parameters when creating the infrastructure and it will be granted the cluster-admin role for the OpenShift Container Platform environment.

3.1.2. SSH Prerequisites

3.1.2.1. SSH Configuration

The SSH service will be configured during the deployment to allow connections using a public key. To make this work, the same public and private key that were created before will be used for the admin user in the instances. Before beginning the deployment of the Microsoft Azure infrastructure and the deployment of OpenShift Container Platform, the SSHprivate key must be converted into a base64 string. Otherwise, the Microsoft Azure infrastructure cannot be configured manually or automatically.

Note

The reason to convert the private key to a base64 string is to avoid possible issues with multiple lines and carriage return characters

This task is performed locally, on the machine that will run the ARM template. In this example, the private key is converted to base64 encoding, and placed on the clipboard.

  • For RHEL/CentOS:
$ cat ~/.ssh/id_rsa | base64 | tr -d '\n' | xclip -selection clipboard
  • For OSX:
$ cat ~/.ssh/id_rsa | base64 | tr -d '\n' | pbcopy
  • For Windows use a Base 64 Encoder app or website to perform the conversion

3.1.3. Red Hat Subscription Prerequisites

A Red Hat account is required to use this template. This account must have appropriate subscriptions available in the account in order to use the template. Either a username and password may be supplied, or an organization ID and activation key.

To be able to access the RPM packages the Pool ID for the subscriptions is required. The next section will show how to locate the Pool ID.

3.1.3.1. Red Hat Subscription Pool ID

In order to assign a subscription, a Pool ID is required. Using a system that currently has a valid subscription to Red Hat products, by using subscription-manager the Pool ID can be acquired. Perform the following command, and take note of the Pool ID.

# subscription-manager list --available

+-------------------------------------------+
    Available Subscriptions
+-------------------------------------------+
Subscription Name:   Red Hat OpenShift Container Platform, Premium (1-2 Sockets)
Provides:            Red Hat OpenShift Enterprise Application Node
                     Red Hat OpenShift Container Platform
                     Red Hat OpenShift Enterprise Client Tools
                     Red Hat Enterprise Linux Server
                     Red Hat Enterprise Linux Atomic Host
                     Red Hat OpenShift Enterprise Infrastructure
                     Red Hat CloudForms
SKU:                 MCT2862
Contract:            38154141
Pool ID:             1b152951269116311123bc522341e1
Provides Management: No
Available:           64
Suggested:           1
Service Level:       Premium
Service Type:        L1-L3
Subscription Type:   Stackable
Ends:                25/08/2017
System Type:         Virtual
Note

The Pool ID is also available in the Subscriptions section of the Red Hat Customer Portal, by selecting the appropriate subscription that will open a detailed view of the subscription, including the Pool ID

3.1.4. Organization ID and Activation Key

Instead using username/password combination, an activation key and organization ID can be used. The activation key is a simple string, chosen by the end-user that contains the subscriptions the user attaches to it to avoid using passwords in the registration procedure. The organization ID is the Red Hat ID for the customer organizations and it is required when using activation keys to identify the activation keys within the organization.

Perform the following steps to obtain the organization ID and activation key.

3.1.4.1. Generating Activation Key

  1. Go to the Red Hat Customer Portal located in http://access.redhat.com and login with your Red Hat username and password. Select [Subscriptions] section in the top left corner then the tab "Activation keys"
  2. Create your key selecting the "New Activation Key" button

Once in there give your key a name, set service level to standard or premium, and add the subscriptions containing OpenShift Container Platform.

3.1.4.2. Getting the Organization ID

In order to find the organization ID, a RHEL server is required. Perform the following on a existing RHEL server.

$ subscription-manager identity
system identity: 8c85ab6d-495c-451e-b38e-5f09bc3342a0
name: bastion
org name: 6616399
org ID: 6616399

Find the value labeled org ID: and save this somewhere for use during the deployment.

3.1.4.3. Using the Organization ID and Activation Key

When running the ARM template, use the organization ID as the RHN User Name, and use the activation key as the RHN Password.

3.1.5. Azure Active Directory Credentials

It is required to create a Microsoft Azure service principal in order to be able to deploy the infrastructure using the ARM template. The service principal object defines the policy and permissions for an application’s use in a specific tenant, providing the basis for a security principal to represent the application at run-time.

Note

For more information about the service principal objects in Azure Active Directory see Use Azure CLI to create a service principal to access resources

In order to create the Azure Active Directory (AAD) client id, and password, the Node.js Azure CLI package is required.

Follow this instructions to create it in a RHEL/CentOS/Fedora system:

  1. Install Node.js

    $ sudo yum -y install npm
  2. Install the Azure CLI Node.js package:

    $ sudo npm install -g azure-cli
  3. Login to Microsoft Azure:

    $ azure login
  4. Create a service principal:

    $ azure ad sp create -n <service_principal_name> -p <password>

    The following is an example output:

    $ azure ad sp create -n openshiftcloudprovider -p Pass@word1
    info:    Executing command ad sp create
    + Creating application openshift demo cloud provider
    + Creating service principal for application 198c4803-1236-4c3f-ad90-46e5f3b4cd2a
    data:    Object Id:               00419334-174b-41e8-9b83-9b5011d8d352
    data:    Display Name:            openshiftcloudprovider
    data:    Service Principal Names:
    data:                             198c4803-1236-4c3f-ad90-46e5f3b4cd2a
    data:                             http://myhomepage
    info:    ad sp create command OK

    Save the Object Id and the Service Principal Names GUID values from the command output.

    • The Object Id will be used to create the role assignment.
    • The Service Principal Names GUID will be used as the aadClientId parameter value (Application ID/Client ID) in the template.
    • The password entered as part of the CLI command will be the aadClientSecret paramter value in the template.
  5. Show the Microsoft Azure account data:

    $ azure account show

    The following is an example output:

    $ azure account show
    info:    Executing command account show
    data:    Name                        : Microsoft Azure Sponsorship
    data:    ID                          : 2581564b-56b4-4512-a140-012d49dfc02c
    data:    State                       : Enabled
    data:    Tenant ID                   : 77ece336-c110-470d-a446-757a69cb9485
    data:    Is Default                  : true
    data:    Environment                 : AzureCloud
    data:    Has Certificate             : Yes
    data:    Has Access Token            : Yes
    data:    User name                   : ssysone@something.com
    data:
    info:    account show command OK

    Save the command output ID value that will be used for the provisioning.

  6. Grant the service principal the access level of contributor to allow OpenShift Container Platform to create/delete resources using the Object ID and ID parameters from the previous steps

    $ azure role assignment create --objectId <objectID> -o contributor -c /subscriptions/<id>/

    The following is an example output:

    # azure role assignment create --objectId 00419334-174b-41e8-9b83-9b5011d8d352 -o contributor -c /subscriptions/2581564b-56b4-4512-a140-012d49dfc02c/
    info:    Executing command role assignment create
    + Finding role with specified name
    /data:    RoleAssignmentId     : /subscriptions/2586c64b-38b4-4527-a140-012d49dfc02c/providers/Microsoft.Authorization/roleAssignments/490c9dd5-0bfa-4b4c-bbc0-aa9af130dd06
    data:    RoleDefinitionName   : Contributor
    data:    RoleDefinitionId     : b24988ac-6180-42a0-ab88-20f7382dd24c
    data:    Scope                : /subscriptions/2586c64b-38b4-4527-a140-012d49dfc02c
    data:    Display Name         : openshiftcloudprovider
    data:    SignInName           : undefined
    data:    ObjectId             : 00419334-174b-41e8-9b83-9b5011d8d352
    data:    ObjectType           : ServicePrincipal
    data:
    +
    info:    role assignment create command OK

3.2. Introduction to the Microsoft Azure Template

Azure Resource Manager templates consist of json files that describes the objects that will be deployed in Microsoft Azure. The main template file for this reference architecture is located in the reference-architecture/azure-ansible/azuredeploy.json file in the git repository. This file is the main ARM template that launches all the other templates under azure-ansible. There are four types of virtual machines created by the template (bastion, master node, infrastructure node and application node) and for each of these types there is a additional json file that defines each VM type.

Virtual Machine typeTemplate file

Bastion

reference-architecture/azure-ansible/bastion.json

Master

reference-architecture/azure-ansible/master.json

Infrastructure node

reference-architecture/azure-ansible/infranode.json

Application node

reference-architecture/azure-ansible/node.json

The ARM template for each type, automatically starts a bash shell script that does part of the initial setup. The main shell script is the reference-architecture/azure-ansible/bastion.sh that handles the generation of the ansible host inventory, as well as the setup and running of ansible across all the hosts. The bastion host also provides isolation of all the hosts in the resource group from the public internet for the purpose of SSH access.

3.3. Alternative Single VM Microsoft Azure Template

In addition to the production template of azuredeploy.json, a single virtual machine version is also available. This template is located at: reference-architecture/azure-ansible/allinone.json This provides for early prototypes and tests of applications in a Red Hat OpenShift Container Platform environment. Note that the single VM does not support the high-availability and load-balancing features of the full azuredeploy.json template. The single virtual machine template only allows you to choose the vm size, and uses a single public ip for console and applications. The Number of Nodes and Wildcard Zone are removed.

3.4. Parameters Required

In order to provision the OpenShift Container Platform environment using the ARM template, the following information is required:

  • A Microsoft Azure subscription, with appropriate core and VM quota limits.
  • Resource Group - Used as the name of the OpenShift Container Platform Cluster - All the assets of a single cluster use the Azure Resource Group to organize and group the assets. This name needs to be unique for each cluster per Azure Region (Location). As some resources will be created using the resource group name, it must be 3-18 characters in length and use numbers and lower-case letters only.
  • Admin username and password - This will be the admin user, used for multiple purposes:

    1. As the SSH user to be able to connect to the bastion host, and administer the cluster.
    2. As an OpenShift Container Platform administrative user, able to create and control OpenShift Container Platform from the command line, or the user interface.
  • SSH Key Data - This is the public key (~/.ssh/id_rsa.pub), generated for the user that will be used to SSH access to all the VMs. During the creation and installation of OpenShift Container Platform virtual machines, the key will automatically be added to each host. This assures proper security and access. This key must be backed up, as its the only principal way to access the cluster for administration.
  • SSH Private Data - This is the private key ~/.ssh/id_rsa file contents that has been base64 encoded. During the creation and installation of OpenShift Container Platform virtual machines, the key will automatically be added to each host. This data should be backed up.
  • Wildcard Zone - Subdomain for applications in the OpenShift Container Platform cluster (required by the load balancer, but nip.io will be used). It is just the subdomain, not the full FQDN. Example wildcardzone parameter will be "refarchapps" and the FQDN will be created as refarchapps.<region>.cloudapp.azure.com
  • Number of Nodes - The template supports the creation of 3 to 30 nodes during greenfield creation of a cluster. Note that the quota of the Microsoft Azure account must support the number chosen.
  • Image - The template supports RHEL (Red Hat Enterprise Linux) 7.3 or later. The image will be upgraded during the installation process to the latest release.
  • Master VM Size (default: Standard_DS4_v2) - The default value gives 8 CPU Cores, 28 Gigabytes of memory, with 56 GB of premium storage local disk. This is used for OpenShift Container Platform master nodes, as well as the bastion host.
  • Infranode VM Size (default: Standard_DS4_v2) - The default value gives 8 CPU Cores, 28 Gigabytes of memory, with 56 GB of premium storage local disk. Infrastructure nodes run the OpenShift Container Platform routers and the OpenShift Container Platform registry pods. As the infrastructure nodes provide the ingress for all applications, its recommended that DS2 be the smallest node used for the infrastructure nodes.
  • Node VM Size (default: Standard_DS4_v2) - The default value gives 8 CPU Cores, 28 Gigabytes of memory, with 56 GB of premium storage local disk. Application nodes is where the application containers run.
  • RHN Username - This should be the username used for the Red Hat Subscription Account that has OpenShift Container Platform entitlements, or the Organization ID if using activation keys.
  • RHN Password - This should be the password for the Red Hat Subscription Account, or the activation key if using activation keys.
  • Subscription Pool ID - This is a number sequence that uniquely identifies the subscriptions that are to be used for the OpenShift Container Platform installation.
  • AAD Client Id - This gives OpenShift Container Platform the Active Directory ID, needed to be able to create, move and delete persistent volumes.
  • AAD Client Secret - The Active Directory Password to match the AAD Client ID. Required to create persistent volumes.
  • OpenShiftSDN - The SDN plugin to be used in the environment (ovs-multitenant by default)
  • Metrics - Deploy Red Hat OpenShift Container Platform metrics components (true by default)
  • Logging - Deploy Red Hat OpenShift Container Platform aggregated logging components (true by default)
  • OPS Logging - Deploy Red Hat OpenShift Container Platform ops aggregated logging components (false by default)

3.5. Provision OpenShift Container Platform environment

There are two ways to provision the OpenShift Container Platform environment. Using a Web Interface, by filling out a form generated by the template for the needed parameters. And the alternate way, is using a ansible playbook to deploy the cluster. The ansible method is ideal when you wish to deploy clusters in a repeatable way, or when you wish to have more than one cluster.

3.5.1. Provisioning ARM Template by using the Web Interface

With the above information ready, go to https://github.com/openshift/openshift-ansible-contrib/tree/master/reference-architecture/azure-ansible and click the [Deploy To Azure] button near the bottom of the page. This will then show the form, to allow the deployment to be started.

Figure 3.1. ARM Template

ARM Template

3.5.2. Provisioning ARM Template by using Ansible

The ARM templates may be deployed via Ansible playbook when you wish to have a repeatable method of creating a cluster, or you wish to create multiple clusters.

In the reference scripts, a additional directory is provided, ansibledeployocp. This provides example playbooks to directly create clusters using Ansible in a Linux environment.

First, install git and ansible.

$ sudo yum -y install ansible git

Then clone the openshift-ansible-contrib repository to a RHEL based host.

$ git clone https://github.com/openshift/openshift-ansible-contrib
$ cd openshift-ansible-contrib/reference-architecture/azure-ansible/ansibledeployocp/

Next, install the dependencies using the prepare playbook:

$ ansible-playbook playbooks/prepare.yml

Microsoft Azure credentials needs to be stored in a file at ~/.azure/credentials with the following format (do not use quotes or double quotes):

[default]
subscription_id=00000000-0000-0000-0000-000000000000
tenant=11111111-1111-1111-1111-111111111111
client_id=33333333-3333-3333-3333-333333333
secret=ServicePrincipalPassword

Where subscription_id and tenant parameters can be obtained from the Microsoft Azure cli:

$ sudo yum install -y nodejs
$ sudo npm install -g azure-cli
$ azure login
$ azure account show
info:    Executing command account show
data:    Name                        : Acme Inc.
data:    ID                          : 00000000-0000-0000-0000-000000000000
data:    State                       : Enabled
data:    Tenant ID                   : 11111111-1111-1111-1111-111111111111
data:    Is Default                  : true
data:    Environment                 : AzureCloud
data:    Has Certificate             : Yes
data:    Has Access Token            : Yes
data:    User name                   : youremail@yourcompany.com
data:
info:    account show command OK

The client_id is the "Service Principal Name" parameter when you create the serviceprincipal:

$ azure ad sp create -n azureansible -p ServicePrincipalPassword

info:    Executing command ad sp create
+ Creating application ansiblelab
+ Creating service principal for application 33333333-3333-3333-3333-333333333
data:    Object Id:               44444444-4444-4444-4444-444444444444
data:    Display Name:            azureansible
data:    Service Principal Names:
data:                             33333333-3333-3333-3333-333333333
data:                             http://azureansible
info:    ad sp create command OK

The secret is the serviceprincipal password.

Ansible Parameters required

The ansible playbook needs some parameters to be specified. There is a vars.yaml example file included in this repository that should be customized with your environment data.

PARAMETERS

$ cp vars.yaml.example vars.yaml
$ vim vars.yaml
  • sshkeydata id_rsa.pub content
  • sshprivatedata id_rsa content in base64 without \n characters (cat ~/.ssh/id_rsa | base64 | tr -d '\n')
  • adminusername User that will be created to login via ssh and as Red Hat OpenShift Container Platform cluster-admin
  • adminpassword Password for the user created (in plain text)
  • rhsmusernamepasswordoractivationkey This should be "usernamepassword" or "activationkey". If "usernamepassword", then the username and password should be specified If "activationkey", then the activation key and organization id should be specified
  • rhnusername The RHN username where the instances will be registered or "activationkey" if activation key method has been chosen
  • rhnpassword The RHN password where the instances will be registered in plain text
  • rhnpassword "organizationid" if activation key method has been chosen else password.
  • subscriptionpoolid The subscription pool id the instances will use
  • resourcegroupname The Microsoft Azure resource name that will be created
  • aadclientid Active Directory ID needed to be able to create, move and delete persistent volumes
  • aadclientsecret The Active Directory Password to match the AAD Client ID
  • wildcardzone Subdomain for applications in the OpenShift cluster (required by the load balancer, but nip.io will be used). It is just the subdomain, not the full FQDN.

Optional (default values are set in playbooks/roles/azure-deploy/default/main.yaml)

  • templatelink - The ARM template that will be deployed
  • numberofnodes - From 3 to 30 nodes
  • image - The operating system image that will be used to create the instances
  • mastervmsize - Master nodes VM size
  • infranodesize - Infrastructure nodes VM size
  • nodevmsize - Application nodes VM size
  • location - westus by default
  • openshiftsdn - "redhat/openshift-ovs-multitenant" by default
  • metrics - Deploy Red Hat OpenShift Container Platform metrics components (true by default)
  • logging - Deploy Red Hat OpenShift Container Platform aggregated logging components (true by default)
  • opslogging - Deploy Red Hat OpenShift Container Platform ops aggregated logging components (false by default)

Running the deploy

$ ansible-playbook -e @vars.yaml playbooks/deploy.yaml

PLAY [localhost] 

TASK [Destroy Azure Deploy]  changed: [localhost] TASK [Destroy Azure Deploy] 
ok: [localhost]

TASK [Create Azure Deploy] 
changed: [localhost]

PLAY RECAP **
localhost                  : ok=3    changed=2    unreachable=0    failed=0

3.6. Post Deployment

Once the playbooks have successfully completed the next steps will be to perform the steps defined in Chapter 4, Operational Management. In the event that OpenShift Container Platform failed to install, follow the steps in Appendix C: Chapter 8, Installation Failure to restart the installation of OpenShift Container Platform.

3.7. Post Provisioning Results

At this point the infrastructure and Red Hat OpenShift Container Platform have been deployed. Log into the Azure web console and check the resources. In the Azure web console, check for the following resources:

  • 3 master nodes
  • 3 infrastructure nodes
  • 3 or more application nodes
  • 1 unique virtual network
  • 3 public IPs
  • 10 network interfaces
  • 4 network security groups
  • 5 storage accounts
  • 2 load balancer profiles
  • 2 load balancer DNS entries
  • 3 routers
  • 3 registries

After the Azure Resource Manager template is submitted, and the ARM deployment succeeds, the ansible install is started automatically.

A wildcard DNS entry must be created if a custom domain is required, by default the nip.io service is be used as explained in the Microsoft Azure DNS section.

Note

When installing using this method the browser certificate must be accepted three times due to the number of masters in the cluster. Failure to accept the certificate can cause disconnect issues and the appearance of network failures.

assetlist