Chapter 2. Components and Configuration

Within this reference environment, the instances are deployed in multiple availability zones in the us-east-1 region by default. Although the default region can be changed, the reference architecture deployment can only be used in Regions with three or more availability zones. The master instances for the OpenShift environment are m4.xlarge and contain three extra disks used for Docker storage, OpenShift ephemeral volumes, and ETCD. The node instances are t2.large and contain two extra disks used for Docker storage and OpenShift ephemeral volumes. The bastion host is a t2.micro. Instance sizing can be changed in the variable files for each installer which is covered in later chapters.

Three load balancers are used in the reference environment. The table below describes the load balancer DNS name, the instances in which the ELB is attached, and the port monitored by the load balancer to state whether an instance is in or out of service.

Both the internal-openshift-master, and the openshift-master ELB utilize the OpenShift Master API port for communication. The internal-openshift-master ELB uses the private subnets for internal cluster communication with the API in order to be more secure. The openshift-master ELB is used for externally accessing the OpenShift environment through the API or the web interface. The openshift-master ELB uses the public subnets to allow communication from anywhere over port 443. The *.apps ELB uses the public subnets and maps to infrastructure nodes. The infrastructure nodes run the router pod which then directs traffic directly from the outside world into OpenShift pods with external routes defined.

The following tables provide the installed software versions for the different servers that make up the Red Hat OpenShift highly available reference environment.

A subscription to the following channels is required in order to deploy this reference environment’s configuration.

The reference architecture environment must be deployed in a Region containing at least 3 availability zones and have 2 free elastic IPs. The environment requires 3 public and 3 private subnets. The usage of 3 public and 3 private subnets allows for the OpenShift deployment to be highly-available and only exposes the required components externally. The subnets can be created during the installation of the reference architecture environment deployment.

The deployment of OpenShift requires a user that has the proper permissions by the AWS IAM administrator. The user must be able to create accounts, S3 buckets, roles, policies, Route53 entries, and deploy ELBs and EC2 instances. It is helpful to have delete permissions in order to be able to redeploy the environment while testing.

An AWS VPC provides the ability to set up custom virtual networking which includes subnets, IP address ranges, route tables and gateways. In this reference implementation guide, a dedicated VPC is created with all its accompanying services to provide a stable network for the OpenShift v3 deployment.

A VPC is created as a logical representation of a networking environment in the AWS cloud. The following subnets and CIDR listed below are used. Substitute the values to ensure no conflict with an existing CIDR or subnet in the environment. The values are defined in /home/<user>/git/openshift-ansible-contrib/reference-architecture/aws-ansible/playbooks/vars/main.yaml.

The VPC is created and a human readable tag is assigned. Six subnets are created and tagged in the VPC. Three subnets are considered public and three subnets are private. The design of one public and one private subnet per Availability Zone allows for high availability(HA). The public subnets are used for the bastion instance and the two external ELBs. The bastion instance is part of the public subnet due to its role as the SSH jumpbox. The two external ELBs allow access to the OpenShift master and the routing of application traffic. In the public route table, an internet gateway and routes are defined and attached to the VPC. The route table has a destination internet gateway associated so that traffic can exit the VPC. The private subnets use the NAT Gateway to communicate to the internet for packages, container images, and Github repositories. The private subnets are assigned their own route table with the NAT Gateway defined. The master, infrastructure, and application nodes are in the private network, as well as the internal-openshift-master, which ensures the nodes cannot be accessed externally.

For more information see https://aws.amazon.com/vpc/

The reference architecture deployment utilizes the AWS NAT Gateway Service to ensure that instances in the private subnets have the ability to download packages, container images, and Github repositories. The NAT Gateway Service funnels all external traffic from private subnets to the outside world. This allows for a smaller external footprint and does not use unneeded public IP and public DNS entries.

In this reference architecture, eight groups are created. The purpose of the security groups is to restrict traffic from outside of the VPC to servers inside of the VPC . The security groups also are used to restrict server to server communications inside the VPC. Security groups provide an extra layer of security similar to a firewall. In the event a port is opened on an instance, the security group will not allow the communication to the port unless explicitly stated in a security group. See the tables below for details on each security group.

2.9.1. Master ELB Security Group

The Master ELB security group allows inbound access on port 443 from the internet to the ELB. The traffic is then allowed to be forwarded to the master instances. See Figure 2.1, “AWS Master ELB Security Group Details - Inbound” diagram and Table 2.5, “AWS Master ELB Security Group Details - Inbound” table below.

Figure 2.1. AWS Master ELB Security Group Details - Inbound

Table 2.5. AWS Master ELB Security Group Details - Inbound

Inbound	From
443 / TCP	Anywhere

Table 2.6. AWS Master ELB Security Group Details - Outbound

Outbound	To
443	ose_master_sg

2.9.2. Internal Master ELB Security Group

The Internal Master ELB is in the private subnet and utilizes the NAT Gateway. Traffic external from the VPC cannot access the Internal Master ELB.

Figure 2.2. AWS Internal Master ELB Security Group Details - Inbound

Inbound	From
443 / TCP	ose_node_sg
443 / TCP	ose_master_sg

Table 2.7. AWS Internal Master ELB Security Group Details - Outbound

Outbound	To
443	ose_master_sg

2.9.3. Bastion Security Group

The bastion security group allows inbound port SSH traffic from outside the VPC. Any connectivity via SSH to the master, application or infrastructure nodes must go through the bastion host. Ensure the bastion host is secured per your companies security requirements.

Table 2.8. AWS Bastion Security Group Details - Inbound

Inbound	From
22 / TCP	Anywhere

Table 2.9. AWS Bastion Security Group Details - Outbound

Outbound	To
All	All

2.9.4. Master Security Group

The master security group allows traffic to the master instances from the two ELBs and nodes to contact the OpenShift API and DNS.

Figure 2.3. AWS Master Security Group Details - Inbound

Inbound	From
8053 / TCP	ose_node_sg
8053 / UDP	ose_node_sg
443 / TCP	ose_internal_elb_master_sg
443 / TCP	ose_elb_master_sg
443 / TCP	ose_node_sg
443 / TCP	ose_master_sg

Table 2.10. AWS Master Security Group Details - Outbound

Outbound	To
All	All

2.9.5. ETCD Security Group

The ETCD security group allows for the ETCD service running on the master instances to reach a quorum. The security group allows for the ose-master-sg to communication with the ETCD for the OpenShift master services.

Table 2.11. ETCD Security Group Details - Inbound

Inbound	From
2379 / TCP	ose-etcd-sg
2379 / TCP	ose-master-sg
2380 / TCP	ose-etcd-sg

Table 2.12. ETCD Security Group Details - Outbound

Outbound	To
All	All

2.9.6. Router ELB Security Group

The Router ELB security group allows inbound access on port 80 and 443. If the applications running on the OpenShift cluster are using different ports this can be adjusted as needed.

Figure 2.4. AWS Router ELB Security Group Details - Inbound

Inbound	From
443 / TCP	Anywhere
80 / TCP	Anywhere

Table 2.13. AWS Router ELB Security Group Details - Outbound

Outbound	To
80	ose_infra_node_sg
443	ose_infra_node_sg

2.9.7. Infrastructure Nodes Security Group

The infrastructure nodes security group allows traffic from the router security group.

Figure 2.5. AWS Infrastructure Nodes Security Group Details - Inbound

Inbound	From
80 / TCP	ose_router_sg
443 / TCP	ose_router_sg

Table 2.14. AWS Infrastructure Nodes Security Group Details - Outbound

Outbound	To
All	All

2.9.8. Nodes Security Group

The node security group only allows traffic from the bastion and traffic relating to OpenShift node services.

Figure 2.6. AWS Nodes Security Group Details - Inbound

Inbound	From
22 / TCP	bastion_sg
10250 / TCP	ose_master_sg
10250 / TCP	ose_node_sg
4789 / UDP	ose_node_sg

Table 2.15. AWS Application Nodes Security Group Details - Outbound

Outbound	To
All	All

DNS is an integral part of a successful OpenShift Compute Platform deployment/environment. AWS has a DNS web service, per Amazon; "Amazon Route 53 is a highly available and scalable cloud DNS web service. It is designed to give developers and businesses an extremely reliable and cost effective way to route end users to internet applications by translating names like www.example.com into numeric IP addresses like 192.0.2.1 that computers use to connect to each other."

OpenShift Compute Platform requires a properly configured wildcard DNS zone that resolves to the IP address of the OpenShift router. For more information, please refer to the Configuring A DNS Wildcard. In this reference architecture Route53 will manage DNS records for the OpenShift Container Platform environment.

For more information see https://aws.amazon.com/route53/

Amazon Machine Images (AMIs) provide the required information to launch an instance. In this guide, the gold image provided by Red Hat is used. The AMI is shared to a specific AWS account which is priced less than the Red Hat Enterprise Linux image provided by AWS.

For more information see AWS Documentation.

AWS provides IAM to securely control access to AWS services and resources for users. IAM can allow or deny access to certain resources for user accounts and for roles within the AWS environment. For this reference architecture, an IAM account will need access to create roles, instances, Route53 entries, ELBs, and many more components. The predefined policy AdministratorAccess has been proven to provide all of the access required to create the environment defined in the this document.

During the installation of OpenShift Container Platform, one account is automatically created to manage a S3 bucket used for the registry. A role and policy are also created to allow for attaching and detaching of EBS volumes for persistent storage within the environment.

For more information see https://aws.amazon.com/iam/

Ansible relies on inventory files and variables to perform playbook runs. As part of the reference architecture provided Ansible playbooks, the inventory is created automatically using a dynamic inventory script. The dynamic inventory script provided queries the Amazon API to display information about EC2 instances. The dynamic inventory script is also referred to as an Ansible Inventory script and the AWS specific script is written in python. The script can manually be executed to provide information about the environment but for this reference architecture, it is automatically called to generate the Ansible Inventory. For the OpenShift installation, the python script and the Ansible module add_host allow for instances to be grouped based on their purpose to be used in later playbooks. The reason the instances can be grouped is because during Phase 1 when the infrastructure was provisioned AWS EC2 tags were applied to each instance. The masters were assigned the master tag, the infrastructure nodes were assigned the infra tag, and the application nodes were assigned the app tag.

For more information see For more information see http://docs.ansible.com/ansible/intro_dynamic_inventory.html

As shown in the Figure 2.7, “Bastion Diagram” the bastion server in this reference architecture provides a secure way to limit SSH access to the AWS environment. The master and node security groups only allow for SSH connectivity between nodes inside of the Security Group while the bastion allows SSH access from everywhere. The bastion host is the only ingress point for SSH in the cluster from external entities. When connecting to the OpenShift Container Platform infrastructure, the bastion forwards the request to the appropriate server. Connecting through the bastion server requires specific SSH configuration. The .ssh/config is outlined in the deployment section of the reference architecture guide.

Figure 2.7. Bastion Diagram

Nodes are AWS instances that serve a specific purpose for OpenShift. OpenShift masters are also considered nodes. Nodes deployed on AWS can be vertically scaled before or after the OpenShift installation using the AWS EC2 console. All OpenShift specific nodes are assigned an IAM role which allows for cloud specific tasks to occur against the environment such as adding persistent volumes or removing a node from the OpenShift Container Platform cluster automatically. There are three types of nodes as described below.

OpenShift uses the Kubernetes concept of a pod, which is one or more containers deployed together on one host, and the smallest compute unit that can be defined, deployed, and managed. For example, a pod could be just a single php application connecting to a database outside of the OpenShift environment or a pod could be a php application that has an ephemeral database. OpenShift pods have the ability to be scaled at runtime or at the time of launch using the OpenShift console or the oc CLI tool. Any container running in the environment is considered a pod. The pods containing the OpenShift router and registry are required to be deployed in the OpenShift environment.

Pods inside of an OpenShift cluster are only reachable via their IP addresses on the cluster network. An edge load balancer can be used to accept traffic from outside networks and proxy the traffic to pods inside the OpenShift cluster.

An OpenShift administrator can deploy routers in an OpenShift cluster. These enable routes created by developers to be used by external clients.

OpenShift routers provide external hostname mapping and load balancing to services over protocols that pass distinguishing information directly to the router; the hostname must be present in the protocol in order for the router to determine where to send it. Routers support the following protocols:

The router utilizes the wildcard zone specified during the installation and configuration of OpenShift. This wildcard zone is used by the router to create routes for a service running within the OpenShift environment to a publically accessible URL. The wildcard zone itself is a wildcard entry in Route53 which is linked using a CNAME to an ELB which performs a health check and forwards traffic to router pods on port 80 and 443.

OpenShift can build Docker images from your source code, deploy them, and manage their lifecycle. To enable this, OpenShift provides an internal, integrated Docker registry that can be deployed in your OpenShift environment to manage images.

The registry stores Docker images and metadata. For production environment, you should use persistent storage for the registry, otherwise any images anyone has built or pushed into the registry would disappear if the pod were to restart.

Using the installation methods described in this document the registry is deployed using a S3 bucket. The S3 bucket allows for multiple pods to be deployed at once for HA but also use the same persistent backend storage. S3 is object based storage which does not get assigned to nodes in the same way that EBS volumes are attached and assigned to a node. The bucket does not mount as block based storage to the node so commands like fdisk or lsblk will not show information in regards to the S3 bucket. The configuration for the S3 bucket and credentials to login to the bucket are stored as OpenShift secrets and applied to the pod. The registry can be scaled to many pods and even have multiple instances of the registry running on the same host due to the use of S3.

OpenShift has the ability to gather metrics from kubelet and store the values in Heapster. OpenShift Cluster Metrics provide the ability to view CPU, memory, and network-based metrics and display the values in the user interface. These metrics can allow for the horizontal autoscaling of pods based on parameters provided by an OpenShift user. It is important to understand capacity planning when deploying metrics into an OpenShift environment.

As best practices when metrics are deployed, persistent storage should be used to allow for metrics to be preserved. Node selectors should be used to specify where the Metics components should run. In the reference architeture environment, the components are deployed on nodes with the label of "role=infra".

openshift_hosted_metrics_deploy: true
openshift_hosted_metrics_storage_kind: dynamic
openshift_hosted_metrics_storage_volume_size: 10Gi
openshift_metrics_hawkular_nodeselector: {"role":"infra"}
openshift_metrics_cassandra_nodeselector: {"role":"infra"}
openshift_metrics_heapster_nodeselector: {"role":"infra"}

OpenShift allows the option to deploy aggregate logging for containers running in the OpenShift environment. OpenShift uses the EFK stack to collect logs from applications and present them to an OpenShift users. Cluster administrators can view all logs, but application developers can only view logs for projects they have permission to view. The EFK stack consists of the following components:

Elasticsearch - object store where logs are stored and provides the ability to search logs Fluentd - gathers and sends logs to Elasticsearch Kibana - a web interface to allow for easy interaction with Elasticsearch Curator - schedule Elasticsearch maintenance operations automatically

Below is an example of some of the best practices when deploying OpenShift logging. Elasticsearch, Kibana, and Curator are deployed on nodes with the label of "role=infra". Specifying the node label ensures that the Elasticsearch and Kibana components are not competing with applications for resources. The Elasticsearch cluster size (openshift_logging_es_cluster_size) of 3 is the minimum number of nodes to ensure data durability and redundancy. The settings below would be defined in a variable file or static inventory.

openshift_hosted_logging_deploy: true
openshift_logging_es_pvc_dynamic: true
openshift_logging_es_pvc_size: 100Gi
openshift_logging_es_cluster_size: 3
openshift_logging_es_nodeselector: {"role":"infra"}
openshift_logging_kibana_nodeselector: {"role":"infra"}
openshift_logging_curator_nodeselector: {"role":"infra"}

There are several options when it comes to authentication of users in OpenShift Container Platform. OpenShift can leverage an existing identity provider within an organization such as LDAP or OpenShift can use external identity providers like GitHub, Google, and GitLab. The configuration of identification providers occurs on the OpenShift master instances. OpenShift allows for multiple identity providers to be specified. The reference architecture document uses GitHub as the authentication provider but any of the other mechanisms would be an acceptable choice. Roles can be added to user accounts to allow for extra privileges such as the ability to list nodes or assign persistent storage volumes to a project.

For more information on GitHub Oauth and other authentication methods see the OpenShift documentation.