Chapter 2. Components and Configuration

This chapter describes the highly available OpenShift Container Platform 3 reference architecture environment on Amazon Web Services (AWS) that is deployed.

The image below provides a high-level representation of the components within this reference architecture. Using Amazon Web Services (AWS), resources are highly available using a combination of multiple availability zones, Elastic Load Balancers(ELB), and an S3 bucket. Instances deployed are given specific roles to support OpenShift. The Bastion host limits the external access to internal servers by ensuring that all SSH traffic passes through the Bastion host. The master instances host the OpenShift master components such as etcd and the OpenShift API. The Application instances are for users to deploy their containers while the Infrastructure instances are used for the OpenShift router and registry. Authentication is managed by GitHub OAuth. OpenShift on AWS has two cloud native storage options; Elastic Block Storage is used for the filesystem of instances but can also be used for persistent storage in containers. The other storage option is S3 which is object based storage. S3 is used for the persistent storage of the OpenShift registry. The network is configured to leverage three AWS ELBs for access to the OpenShift API, OpenShift console, and the OpenShift routers. The first ELB is for the OpenShift API and console access originating from outside of the cluster. The second ELB is for API access within the cluster. The third ELB is for accessing services deployed in the cluster that have been exposed through routes. Finally, the image shows that DNS is handled by Route53. In this case the systems engineering team is managing all DNS entries through Route53.

OSE on AWS Architecture

This reference architecture breaks down the deployment into separate phases.

  • Phase 1: Provision the infrastructure on AWS
  • Phase 2: Provision OpenShift Container Platform on AWS
  • Phase 3: Post deployment activities

For Phase 1, the provisioning of the environment is done using a series of Ansible playbooks that are provided in the openshift-ansible-contrib github repo. Once the infrastructure is deployed the playbooks will flow automatically into Phase 2. Phase 2 is the installation of OpenShift Container Platform which is done via Ansible playbooks. These playbooks are installed by the openshift-ansible-playbooks rpm package. The playbooks in openshift-ansible-contrib utilize the playbooks provided by the openshift-ansible-playbooks RPM package to perform the installation of OpenShift and also to configure AWS specific parameters. During Phase 2 the router and registry are deployed. The last phase, Phase 3, concludes the deployment by confirming the environment was deployed properly. This is done by running tools like oadm diagnostics and the systems engineering teams validation Ansible playbook.

Note

The scripts provided in the github repo are not supported by Red Hat. They merely provide a mechanism that can be used to build out your own infrastructure.

2.1. Elastic Compute Cloud Instance Details

Within this reference environment, the instances are deployed in multiple availability zones in the us-east-1 region by default. Although the default region can be changed, the reference architecture deployment can only be used in Regions with three or more availability zones. The master instances for the OpenShift environment are m4.xlarge and contain three extra disks used for Docker storage, OpenShift ephemeral volumes, and ETCD. The node instances are t2.large and contain two extra disks used for Docker storage and OpenShift ephemeral volumes. The bastion host is a t2.micro. Instance sizing can be changed in the variable files for each installer which is covered in later chapters.

2.2. Elastic Load Balancers Details

Three load balancers are used in the reference environment. The table below describes the load balancer DNS name, the instances in which the ELB is attached, and the port monitored by the load balancer to state whether an instance is in or out of service.

Table 2.1. Elastic Load Balancers

ELBAssigned InstancesPort

openshift-master.sysdeseng.com

master01-3

443

internal-openshift-master.sysdeseng.com

master01-3

443

*.apps.sysdeseng.com

infra-nodes01-3

80 and 443

Both the internal-openshift-master, and the openshift-master ELB utilize the OpenShift Master API port for communication. The internal-openshift-master ELB uses the private subnets for internal cluster communication with the API in order to be more secure. The openshift-master ELB is used for externally accessing the OpenShift environment through the API or the web interface. The openshift-master ELB uses the public subnets to allow communication from anywhere over port 443. The *.apps ELB uses the public subnets and maps to infrastructure nodes. The infrastructure nodes run the router pod which then directs traffic directly from the outside world into OpenShift pods with external routes defined.

2.3. Software Version Details

The following tables provide the installed software versions for the different servers that make up the Red Hat OpenShift highly available reference environment.

Table 2.2. RHEL OSEv3 Details

SoftwareVersion

Red Hat Enterprise Linux 7.3 x86_64

kernel-3.10.0.x

Atomic-OpenShift{master/clients/node/sdn-ovs/utils}

3.5.x.x

Docker

1.12.x

Ansible

2.2.1.x

2.4. Required Channels

A subscription to the following channels is required in order to deploy this reference environment’s configuration.

Table 2.3. Required Channels - OSEv3 Master and Node Instances

ChannelRepository Name

Red Hat Enterprise Linux 7 Server (RPMs)

rhel-7-server-rpms

Red Hat OpenShift Container Platform 3.5 (RPMs)

rhel-7-server-ose-3.5-rpms

Red Hat Enterprise Linux 7 Server - Extras (RPMs)

rhel-7-server-extras-rpms

Red Hat Enterprise Linux Fast Datapath (RHEL 7 Server) (RPMs)

rhel-7-fast-datapath-rpms

2.5. AWS Region Requirements

The reference architecture environment must be deployed in a Region containing at least 3 availability zones and have 2 free elastic IPs. The environment requires 3 public and 3 private subnets. The usage of 3 public and 3 private subnets allows for the OpenShift deployment to be highly-available and only exposes the required components externally. The subnets can be created during the installation of the reference architecture environment deployment.

2.6. Permissions for Amazon Web Services

The deployment of OpenShift requires a user that has the proper permissions by the AWS IAM administrator. The user must be able to create accounts, S3 buckets, roles, policies, Route53 entries, and deploy ELBs and EC2 instances. It is helpful to have delete permissions in order to be able to redeploy the environment while testing.

2.7. Virtual Private Cloud (VPC)

An AWS VPC provides the ability to set up custom virtual networking which includes subnets, IP address ranges, route tables and gateways. In this reference implementation guide, a dedicated VPC is created with all its accompanying services to provide a stable network for the OpenShift v3 deployment.

A VPC is created as a logical representation of a networking environment in the AWS cloud. The following subnets and CIDR listed below are used. Substitute the values to ensure no conflict with an existing CIDR or subnet in the environment. The values are defined in /home/<user>/git/openshift-ansible-contrib/reference-architecture/aws-ansible/playbooks/vars/main.yaml.

Table 2.4. VPC Networking

CIDR / SubnetValues

CIDR

10.20.0.0/16

Private Subnet 1

10.20.1.0/24

Private Subnet 2

10.20.2.0/24

Private Subnet 3

10.20.3.0/24

Public Subnet 1

10.20.4.0/24

Public Subnet 2

10.20.5.0/24

Public Subnet 3

10.20.6.0/24

The VPC is created and a human readable tag is assigned. Six subnets are created and tagged in the VPC. Three subnets are considered public and three subnets are private. The design of one public and one private subnet per Availability Zone allows for high availability(HA). The public subnets are used for the bastion instance and the two external ELBs. The bastion instance is part of the public subnet due to its role as the SSH jumpbox. The two external ELBs allow access to the OpenShift master and the routing of application traffic. In the public route table, an internet gateway and routes are defined and attached to the VPC. The route table has a destination internet gateway associated so that traffic can exit the VPC. The private subnets use the NAT Gateway to communicate to the internet for packages, container images, and Github repositories. The private subnets are assigned their own route table with the NAT Gateway defined. The master, infrastructure, and application nodes are in the private network, as well as the internal-openshift-master, which ensures the nodes cannot be accessed externally.

For more information see https://aws.amazon.com/vpc/

2.8. NAT Gateway

The reference architecture deployment utilizes the AWS NAT Gateway Service to ensure that instances in the private subnets have the ability to download packages, container images, and Github repositories. The NAT Gateway Service funnels all external traffic from private subnets to the outside world. This allows for a smaller external footprint and does not use unneeded public IP and public DNS entries.

2.9. Security Groups

In this reference architecture, eight groups are created. The purpose of the security groups is to restrict traffic from outside of the VPC to servers inside of the VPC . The security groups also are used to restrict server to server communications inside the VPC. Security groups provide an extra layer of security similar to a firewall. In the event a port is opened on an instance, the security group will not allow the communication to the port unless explicitly stated in a security group. See the tables below for details on each security group.

2.9.1. Master ELB Security Group

The Master ELB security group allows inbound access on port 443 from the internet to the ELB. The traffic is then allowed to be forwarded to the master instances. See Figure 2.1, “AWS Master ELB Security Group Details - Inbound” diagram and Table 2.5, “AWS Master ELB Security Group Details - Inbound” table below.

Figure 2.1. AWS Master ELB Security Group Details - Inbound

Master ELB SG

Table 2.5. AWS Master ELB Security Group Details - Inbound

InboundFrom

443 / TCP

Anywhere

Table 2.6. AWS Master ELB Security Group Details - Outbound

OutboundTo

443

ose_master_sg

2.9.2. Internal Master ELB Security Group

The Internal Master ELB is in the private subnet and utilizes the NAT Gateway. Traffic external from the VPC cannot access the Internal Master ELB.

Figure 2.2. AWS Internal Master ELB Security Group Details - Inbound

Internal Master ELB SG
InboundFrom

443 / TCP

ose_node_sg

443 / TCP

ose_master_sg

Table 2.7. AWS Internal Master ELB Security Group Details - Outbound

OutboundTo

443

ose_master_sg

2.9.3. Bastion Security Group

The bastion security group allows inbound port SSH traffic from outside the VPC. Any connectivity via SSH to the master, application or infrastructure nodes must go through the bastion host. Ensure the bastion host is secured per your companies security requirements.

Bastion SG

Table 2.8. AWS Bastion Security Group Details - Inbound

InboundFrom

22 / TCP

Anywhere

Table 2.9. AWS Bastion Security Group Details - Outbound

OutboundTo

All

All

2.9.4. Master Security Group

The master security group allows traffic to the master instances from the two ELBs and nodes to contact the OpenShift API and DNS.

Figure 2.3. AWS Master Security Group Details - Inbound

Master SG
InboundFrom

8053 / TCP

ose_node_sg

8053 / UDP

ose_node_sg

443 / TCP

ose_internal_elb_master_sg

443 / TCP

ose_elb_master_sg

443 / TCP

ose_node_sg

443 / TCP

ose_master_sg

Table 2.10. AWS Master Security Group Details - Outbound

OutboundTo

All

All

2.9.5. ETCD Security Group

The ETCD security group allows for the ETCD service running on the master instances to reach a quorum. The security group allows for the ose-master-sg to communication with the ETCD for the OpenShift master services.

ETCD SG

Table 2.11. ETCD Security Group Details - Inbound

InboundFrom

2379 / TCP

ose-etcd-sg

2379 / TCP

ose-master-sg

2380 / TCP

ose-etcd-sg

Table 2.12. ETCD Security Group Details - Outbound

OutboundTo

All

All

2.9.6. Router ELB Security Group

The Router ELB security group allows inbound access on port 80 and 443. If the applications running on the OpenShift cluster are using different ports this can be adjusted as needed.

Figure 2.4. AWS Router ELB Security Group Details - Inbound

Router ELB SG
InboundFrom

443 / TCP

Anywhere

80 / TCP

Anywhere

Table 2.13. AWS Router ELB Security Group Details - Outbound

OutboundTo

80

ose_infra_node_sg

443

ose_infra_node_sg

2.9.7. Infrastructure Nodes Security Group

The infrastructure nodes security group allows traffic from the router security group.

Figure 2.5. AWS Infrastructure Nodes Security Group Details - Inbound

Infrastructure Node SG
InboundFrom

80 / TCP

ose_router_sg

443 / TCP

ose_router_sg

Table 2.14. AWS Infrastructure Nodes Security Group Details - Outbound

OutboundTo

All

All

2.9.8. Nodes Security Group

The node security group only allows traffic from the bastion and traffic relating to OpenShift node services.

Figure 2.6. AWS Nodes Security Group Details - Inbound

Node SG
InboundFrom

22 / TCP

bastion_sg

10250 / TCP

ose_master_sg

10250 / TCP

ose_node_sg

4789 / UDP

ose_node_sg

Table 2.15. AWS Application Nodes Security Group Details - Outbound

OutboundTo

All

All

2.10. Route53

DNS is an integral part of a successful OpenShift Compute Platform deployment/environment. AWS has a DNS web service, per Amazon; "Amazon Route 53 is a highly available and scalable cloud DNS web service. It is designed to give developers and businesses an extremely reliable and cost effective way to route end users to internet applications by translating names like www.example.com into numeric IP addresses like 192.0.2.1 that computers use to connect to each other."

OpenShift Compute Platform requires a properly configured wildcard DNS zone that resolves to the IP address of the OpenShift router. For more information, please refer to the Configuring A DNS Wildcard. In this reference architecture Route53 will manage DNS records for the OpenShift Container Platform environment.

For more information see https://aws.amazon.com/route53/

2.10.1. Public Zone

The reference architecture environment automatically adds entries into Route53. A Hosted Zone is required for OpenShift. A domain name can either purchased through AWS or another external provider such as Google Domains or GoDaddy. If using a zone from an external provider ensure the NS records point to a Route53 hosted zone. Steps will be detailed in Chapter 3 of this document.

2.11. Amazon Machine Images

Amazon Machine Images (AMIs) provide the required information to launch an instance. In this guide, the gold image provided by Red Hat is used. The AMI is shared to a specific AWS account which is priced less than the Red Hat Enterprise Linux image provided by AWS.

For more information see AWS Documentation.

2.11.1. Red Hat Gold Image

The Red Hat Cloud Access provides gold images for Atomic and Red Hat Enterprise Linux. Since a subscription is required to install OpenShift then it is not necessary to use the Amazon provided image which has a built in charge back for the RHEL subscription.

To register for the Red Hat Cloud Access Gold Image please see Red Hat’s Website and select the tab for Red Hat Gold Image.

2.12. Identity and Access Management

AWS provides IAM to securely control access to AWS services and resources for users. IAM can allow or deny access to certain resources for user accounts and for roles within the AWS environment. For this reference architecture, an IAM account will need access to create roles, instances, Route53 entries, ELBs, and many more components. The predefined policy AdministratorAccess has been proven to provide all of the access required to create the environment defined in the this document.

During the installation of OpenShift Container Platform, one account is automatically created to manage a S3 bucket used for the registry. A role and policy are also created to allow for attaching and detaching of EBS volumes for persistent storage within the environment.

For more information see https://aws.amazon.com/iam/

2.13. Dynamic Inventory

Ansible relies on inventory files and variables to perform playbook runs. As part of the reference architecture provided Ansible playbooks, the inventory is created automatically using a dynamic inventory script. The dynamic inventory script provided queries the Amazon API to display information about EC2 instances. The dynamic inventory script is also referred to as an Ansible Inventory script and the AWS specific script is written in python. The script can manually be executed to provide information about the environment but for this reference architecture, it is automatically called to generate the Ansible Inventory. For the OpenShift installation, the python script and the Ansible module add_host allow for instances to be grouped based on their purpose to be used in later playbooks. The reason the instances can be grouped is because during Phase 1 when the infrastructure was provisioned AWS EC2 tags were applied to each instance. The masters were assigned the master tag, the infrastructure nodes were assigned the infra tag, and the application nodes were assigned the app tag.

For more information see For more information see http://docs.ansible.com/ansible/intro_dynamic_inventory.html

2.14. Bastion

As shown in the Figure 2.7, “Bastion Diagram” the bastion server in this reference architecture provides a secure way to limit SSH access to the AWS environment. The master and node security groups only allow for SSH connectivity between nodes inside of the Security Group while the bastion allows SSH access from everywhere. The bastion host is the only ingress point for SSH in the cluster from external entities. When connecting to the OpenShift Container Platform infrastructure, the bastion forwards the request to the appropriate server. Connecting through the bastion server requires specific SSH configuration. The .ssh/config is outlined in the deployment section of the reference architecture guide.

Figure 2.7. Bastion Diagram

Bastion Server

2.15. Nodes

Nodes are AWS instances that serve a specific purpose for OpenShift. OpenShift masters are also considered nodes. Nodes deployed on AWS can be vertically scaled before or after the OpenShift installation using the AWS EC2 console. All OpenShift specific nodes are assigned an IAM role which allows for cloud specific tasks to occur against the environment such as adding persistent volumes or removing a node from the OpenShift Container Platform cluster automatically. There are three types of nodes as described below.

2.15.1. Master nodes

The master nodes contain the master components, including the API server, controller manager server and ETCD. The master maintains the clusters configuration, manages nodes in its OpenShift cluster. The master assigns pods to nodes and synchronizes pod information with service configuration. The master is used to define routes, services, and volume claims for pods deployed within the OpenShift environment.

2.15.2. Infrastructure nodes

The infrastructure nodes are used for the router and registry pods. These nodes could be used if the optional components Kibana and Hawkular metrics are required. The storage for the Docker registry that is deployed on the infrastructure nodes is S3 which allows for multiple pods to use the same storage. AWS S3 storage is used because it is synchronized between the availability zones, providing data redundancy.

2.15.3. Application nodes

The Application nodes are the instances where non-infrastructure based containers run. Depending on the application, AWS specific storage can be applied such as a Elastic Block Storage which can be assigned using a Persistent Volume Claim for application data that needs to persist between container restarts. A configuration parameter is set on the master which ensures that OpenShift Container Platform user containers will be placed on the application nodes by default.

2.15.4. Node labels

All OpenShift Container Platform nodes are assigned a label. This allows certain pods to be deployed on specific nodes. For example, nodes labeled infra are Infrastructure nodes. These nodes run the router and registry pods. Nodes with the label app are nodes used for end user Application pods. The configuration parameter 'defaultNodeSelector: "role=app" in /etc/origin/master/master-config.yaml ensures all projects automatically are deployed on Application nodes.

2.16. OpenShift Pods

OpenShift uses the Kubernetes concept of a pod, which is one or more containers deployed together on one host, and the smallest compute unit that can be defined, deployed, and managed. For example, a pod could be just a single php application connecting to a database outside of the OpenShift environment or a pod could be a php application that has an ephemeral database. OpenShift pods have the ability to be scaled at runtime or at the time of launch using the OpenShift console or the oc CLI tool. Any container running in the environment is considered a pod. The pods containing the OpenShift router and registry are required to be deployed in the OpenShift environment.

2.17. Router

Pods inside of an OpenShift cluster are only reachable via their IP addresses on the cluster network. An edge load balancer can be used to accept traffic from outside networks and proxy the traffic to pods inside the OpenShift cluster.

An OpenShift administrator can deploy routers in an OpenShift cluster. These enable routes created by developers to be used by external clients.

OpenShift routers provide external hostname mapping and load balancing to services over protocols that pass distinguishing information directly to the router; the hostname must be present in the protocol in order for the router to determine where to send it. Routers support the following protocols:

  • HTTP
  • HTTPS (with SNI)
  • WebSockets
  • TLS with SNI

The router utilizes the wildcard zone specified during the installation and configuration of OpenShift. This wildcard zone is used by the router to create routes for a service running within the OpenShift environment to a publically accessible URL. The wildcard zone itself is a wildcard entry in Route53 which is linked using a CNAME to an ELB which performs a health check and forwards traffic to router pods on port 80 and 443.

2.18. Registry

OpenShift can build Docker images from your source code, deploy them, and manage their lifecycle. To enable this, OpenShift provides an internal, integrated Docker registry that can be deployed in your OpenShift environment to manage images.

The registry stores Docker images and metadata. For production environment, you should use persistent storage for the registry, otherwise any images anyone has built or pushed into the registry would disappear if the pod were to restart.

Using the installation methods described in this document the registry is deployed using a S3 bucket. The S3 bucket allows for multiple pods to be deployed at once for HA but also use the same persistent backend storage. S3 is object based storage which does not get assigned to nodes in the same way that EBS volumes are attached and assigned to a node. The bucket does not mount as block based storage to the node so commands like fdisk or lsblk will not show information in regards to the S3 bucket. The configuration for the S3 bucket and credentials to login to the bucket are stored as OpenShift secrets and applied to the pod. The registry can be scaled to many pods and even have multiple instances of the registry running on the same host due to the use of S3.

2.19. OpenShift Metrics

OpenShift has the ability to gather metrics from kubelet and store the values in Heapster. OpenShift Cluster Metrics provide the ability to view CPU, memory, and network-based metrics and display the values in the user interface. These metrics can allow for the horizontal autoscaling of pods based on parameters provided by an OpenShift user. It is important to understand capacity planning when deploying metrics into an OpenShift environment.

As best practices when metrics are deployed, persistent storage should be used to allow for metrics to be preserved. Node selectors should be used to specify where the Metics components should run. In the reference architeture environment, the components are deployed on nodes with the label of "role=infra".

openshift_hosted_metrics_deploy: true
openshift_hosted_metrics_storage_kind: dynamic
openshift_hosted_metrics_storage_volume_size: 10Gi
openshift_metrics_hawkular_nodeselector: {"role":"infra"}
openshift_metrics_cassandra_nodeselector: {"role":"infra"}
openshift_metrics_heapster_nodeselector: {"role":"infra"}

2.20. OpenShift Logging

OpenShift allows the option to deploy aggregate logging for containers running in the OpenShift environment. OpenShift uses the EFK stack to collect logs from applications and present them to an OpenShift users. Cluster administrators can view all logs, but application developers can only view logs for projects they have permission to view. The EFK stack consists of the following components:

Elasticsearch - object store where logs are stored and provides the ability to search logs Fluentd - gathers and sends logs to Elasticsearch Kibana - a web interface to allow for easy interaction with Elasticsearch Curator - schedule Elasticsearch maintenance operations automatically

Below is an example of some of the best practices when deploying OpenShift logging. Elasticsearch, Kibana, and Curator are deployed on nodes with the label of "role=infra". Specifying the node label ensures that the Elasticsearch and Kibana components are not competing with applications for resources. The Elasticsearch cluster size (openshift_logging_es_cluster_size) of 3 is the minimum number of nodes to ensure data durability and redundancy. The settings below would be defined in a variable file or static inventory.

Note

Fluentd runs on all OpenShift nodes regardless of node label.

openshift_hosted_logging_deploy: true
openshift_logging_es_pvc_dynamic: true
openshift_logging_es_pvc_size: 100Gi
openshift_logging_es_cluster_size: 3
openshift_logging_es_nodeselector: {"role":"infra"}
openshift_logging_kibana_nodeselector: {"role":"infra"}
openshift_logging_curator_nodeselector: {"role":"infra"}

2.21. Authentication

There are several options when it comes to authentication of users in OpenShift Container Platform. OpenShift can leverage an existing identity provider within an organization such as LDAP or OpenShift can use external identity providers like GitHub, Google, and GitLab. The configuration of identification providers occurs on the OpenShift master instances. OpenShift allows for multiple identity providers to be specified. The reference architecture document uses GitHub as the authentication provider but any of the other mechanisms would be an acceptable choice. Roles can be added to user accounts to allow for extra privileges such as the ability to list nodes or assign persistent storage volumes to a project.

For more information on GitHub Oauth and other authentication methods see the OpenShift documentation.