Chapter 2. Red Hat OpenShift Container Platform Prerequisites
2.1. Red Hat Cloud Access
AWS provides a RHEL based machine image to use as root volumes for EC2 virtual machines. AWS adds an additional per hour charge to EC2s running the image to cover the price of the license. This enables customers to pay one fee to AWS and not deal with multiple vendors. To other customers this may be a security risk as AWS has the duty of building the machine image and maintaining its lifecycle.
Red Hat provides a bring your own license program for Red Hat Enterprise Linux in AWS. Machine images are presented to participating customer for use as EC2 root volumes. Once signed up Red Hat machine images are then made available in the AMI Private Images inventory.
Red Hat Cloud Access program: https://www.redhat.com/en/technologies/cloud-computing/cloud-access
This reference architecture expects customers to be Cloud Access participants
2.2. Execution environment
RHEL 7 is the only operating system supported by Red Hat OpenShift Container Platform installer therefore provider infrastructure deployment and installer must be run from one of the following locations running RHEL 7:
- Local workstation / server / virtual machine / Bastion
- Jenkins continuous delivery engine
This reference architecture focus’s on deploying and installing Red Hat OpenShift Container Platform from local workstation/server/virtual machine and bastion. Registering this host to Red Hat subscription manager will be a requirement to get access to Red Hat OpenShift Container Platform installer and related rpms. Jenkins is out of scope.
2.3. AWS IAM user
An IAM user with an admin policy and access key is required to interface with AWS API and deploy infrastructure using either AWS CLI or Boto AWS SDK.
Follow this procedure to create an IAM user https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html#id_users_create_console
Follow this procedure to create an access key for an IAM user https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
2.4. EPEL and supporting rpms
epel
repo contains supporting rpms to assist with tooling and Red Hat OpenShift Container Platform infrastructure deployment and installation.
Use the following command to enable the epel
repo:
$ yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
python-pip
is useful to install awscli, boto3/boto AWS python bindings.
$ yum install -y python-pip
jq
simplifies data extraction from json output or files.
$ yum install -y jq
epel
is no longer needed and can be removed.
$ yum erase epel-release
2.5. AWS Command Line Interface (awscli
)
The awscli
can be used to deploy all of the components associated with this Reference Architecture.
Detailed installation procedure is here: https://docs.aws.amazon.com/cli/latest/userguide/installing.html
This reference architecture supports the following minimum version of awscli:
$ aws --version aws-cli/1.15.16 Python/2.7.5 Linux/3.10.0-862.el7.x86_64 botocore/1.8.50
2.6. Boto3 AWS SDK for Python
Boto3 is the AWS SDK for Python, which allows Python developers to write software that makes use of AWS services. Boto provides an easy to use, object-oriented API, and low-level direct service access.
Detailed Boto3 AWS python bindings installation procedure is here: http://boto3.readthedocs.io/en/latest/guide/quickstart.html#installation
Detailed legacy Boto AWS python bindings installation procedure is here: http://boto.cloudhackers.com/en/latest/getting_started.html
If using Ansible to deploy AWS infrastructure installing boto3 AND legacy boto python bindings is mandatory as some Ansible modules still use the legacy boto AWS python bindings.
Ansible AWS tasks can experience random errors due the speed of execution and AWS API rate limiting. Follow this procedure to ensure Ansible tasks complete sucessfully:
$ cat << EOF > ~/.boto [Boto] debug = 0 num_retries = 10 EOF
This reference architecture supports the following minimum version of boto3:
$ pip freeze | grep boto3 boto3==1.5.24
This reference architecture supports the following minimum version of boto:
$ pip freeze | grep boto boto==2.48.0
2.7. Set Up Client Credentials
Once the AWS CLI and Boto AWS SDK are installed, the credential needs to be set up.
Detailed IAM user authorization and authentication credentials procedure is here: https://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html
Shell environment variables can be an alternative to a static credential file.
$ export AWS_ACCESS_KEY_ID = <your aws_access_key_id here> $ export AWS_SECRET_ACCESS_KEY = <your aws_secret_access_key here>
2.8. Client Testing
A successful awscli
and credentials test will look similar to the following:
$ aws sts get-caller-identity output: { "Account": "123123123123", "UserId": "TH75ISMYR3F4RCHUS3R1D", "Arn": "arn:aws:iam::123123123123:user/refarchuser" }
The following error indicates an issue with the AWS IAM user’s access key and local credentials:
$ aws sts get-caller-identity output: An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid.
A successful boto and credentials will look similar to the following:
$ cat << EOF | python import boto3 print(boto3.client('sts').get_caller_identity()['Arn']) EOF output: arn:aws:iam::123123123123:user/refarch
The following error indicates an issue with the AWS IAM user’s access key and local credentials:
output: .... botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid.
2.9. Git (optional)
Git is a fast, scalable, distributed revision control system with an unusually rich command set that provides both high-level operations and full access to internals.
Install git
package
$ yum install -y git
When using Ansible to deploy AWS infrastructure this step is mandatory as git is used to clone a source code repository.
2.10. Subscription Manager
Register instance with Red Hat Subscription Manager and activate yum repos
$ subscription-manager register $ subscription-manager attach \ --pool NUMBERIC_POOLID $ subscription-manager repos \ --disable="*" \ --enable=rhel-7-server-rpms \ --enable=rhel-7-server-extras-rpms \ --enable=rhel-7-server-ansible-2.4-rpms \ --enable=rhel-7-server-ose-3.9-rpms $ yum -y install atomic-openshift-utils
2.11. Create AWS Infrastructure for Red Hat OpenShift Container Platform
Infrastructure creation can be executed anywhere provided the AWS API is network accessible and tooling requirements have been met. It is common for local workstations to be used for this step.
When using a bastion as the installer execution environment be sure that all of the previous requirements get met. In some cases such as when an AWS EC2 is being used as a bastion there can be a chicken / egg issue where AWS network components and bastion instance to be used as the installer execution environment is not yet available.
2.11.1. CLI
The following awscli
commands are provided to expose the complexity behind automation tools such Ansible.
Review the following environment variables now and ensure values fit requirements. When values are satisfactory execute the commands in terminal.
$ export clusterid="rhocp" $ export dns_domain="example.com" $ export region="us-east-1" $ export cidrvpc="172.16.0.0/16" $ export cidrsubnets_public=("172.16.0.0/24" "172.16.1.0/24" "172.16.2.0/24") $ export cidrsubnets_private=("172.16.16.0/20" "172.16.32.0/20" "172.16.48.0/20") $ export ec2_type_bastion="t2.medium" $ export ec2_type_master="m5.2xlarge" $ export ec2_type_infra="m5.2xlarge" $ export ec2_type_node="m5.2xlarge" $ export rhel_release="rhel-7.5"
Create a public private ssh keypair to be used with ssh-agent and ssh authentication on AWS EC2s.
$ if [ ! -f ${HOME}/.ssh/${clusterid}.${dns_domain} ]; then echo 'Enter ssh key password' read -r passphrase ssh-keygen -P ${passphrase} -o -t rsa -f ~/.ssh/${clusterid}.${dns_domain} fi $ export sshkey=($(cat ~/.ssh/${clusterid}.${dns_domain}.pub))
Gather available Availability Zones. The first 3 are used.
$ IFS=$' '; export az=($(aws ec2 describe-availability-zones \ --filters "Name=region-name,Values=${region}" | \ jq -r '.[][].ZoneName' | \ head -3 | \ tr '\n' ' ' | \ sed -e "s/ $//g")); $ unset IFS
Retrive Red Hat Cloud Access AMI
$ export ec2ami=($(aws ec2 describe-images \ --region ${region} --owners 309956199498 | \ jq -r '.Images[] | [.Name,.ImageId] | @csv' | \ sed -e 's/,/ /g' | \ sed -e 's/"//g' | \ grep HVM_GA | \ grep Access2-GP2 | \ grep -i ${rhel_release} | \ sort | \ tail -1))
Deploy IAM users and sleep for 15 sec so that AWS can instantiate the users
$ export iamuser=$(aws iam create-user --user-name ${clusterid}.${dns_domain}-admin) $ export s3user=$(aws iam create-user --user-name ${clusterid}.${dns_domain}-registry) $ sleep 15
Create access key for IAM users
$ export iamuser_accesskey=$(aws iam create-access-key --user-name ${clusterid}.${dns_domain}-admin) $ export s3user_accesskey=$(aws iam create-access-key --user-name ${clusterid}.${dns_domain}-registry)
Create and attach policies to IAM users
$ cat << EOF > ~/.iamuser_policy_cpk { "Version": "2012-10-17", "Statement": [ { "Action": [ "ec2:DescribeVolume*", "ec2:CreateVolume", "ec2:CreateTags", "ec2:DescribeInstances", "ec2:AttachVolume", "ec2:DetachVolume", "ec2:DeleteVolume", "ec2:DescribeSubnets", "ec2:CreateSecurityGroup", "ec2:DescribeSecurityGroups", "ec2:DescribeRouteTables", "ec2:AuthorizeSecurityGroupIngress", "ec2:RevokeSecurityGroupIngress", "elasticloadbalancing:DescribeTags", "elasticloadbalancing:CreateLoadBalancerListeners", "elasticloadbalancing:ConfigureHealthCheck", "elasticloadbalancing:DeleteLoadBalancerListeners", "elasticloadbalancing:RegisterInstancesWithLoadBalancer", "elasticloadbalancing:DescribeLoadBalancers", "elasticloadbalancing:CreateLoadBalancer", "elasticloadbalancing:DeleteLoadBalancer", "elasticloadbalancing:ModifyLoadBalancerAttributes", "elasticloadbalancing:DescribeLoadBalancerAttributes" ], "Resource": "*", "Effect": "Allow", "Sid": "1" } ] } EOF $ aws iam put-user-policy \ --user-name ${clusterid}.${dns_domain}-admin \ --policy-name Admin \ --policy-document file://~/.iamuser_policy_cpk $ cat << EOF > ~/.iamuser_policy_s3 { "Version": "2012-10-17", "Statement": [ { "Action": [ "s3:*" ], "Resource": [ "arn:aws:s3:::${clusterid}.${dns_domain}-registry", "arn:aws:s3:::${clusterid}.${dns_domain}-registry/*" ], "Effect": "Allow", "Sid": "1" } ] } EOF $ aws iam put-user-policy \ --user-name ${clusterid}.${dns_domain}-registry \ --policy-name S3 \ --policy-document file://~/.iamuser_policy_s3 $ rm -rf ~/.iamuser_policy*
Deploy EC2 keypair
$ export keypair=$(aws ec2 import-key-pair \ --key-name ${clusterid}.${dns_domain} \ --public-key-material file://~/.ssh/${clusterid}.${dns_domain}.pub \ )
Deploy S3 bucket and policy
$ export aws_s3bucket=$(aws s3api create-bucket \ --bucket $(echo ${clusterid}.${dns_domain}-registry) \ --region ${region} \ ) $ aws s3api put-bucket-tagging \ --bucket $(echo ${clusterid}.${dns_domain}-registry) \ --tagging "TagSet=[{Key=Clusterid,Value=${clusterid}}]" $ aws s3api put-bucket-policy \ --bucket $(echo ${clusterid}.${dns_domain}-registry) \ --policy "\ { \ \"Version\": \"2012-10-17\", \"Statement\": [ \ { \ \"Action\": \"s3:*\", \ \"Effect\": \"Allow\", \ \"Principal\": { \ \"AWS\": \"$(echo ${s3user} | jq -r '.User.Arn')\" \ }, \ \"Resource\": \"arn:aws:s3:::$(echo ${aws_s3bucket} | jq -r '.Location' | sed -e 's/^\///g')\", \ \"Sid\": \"1\" \ } \ ] \ }"
If region is a region other than us-east-1 then aws s3api create-bucket
will require --create-bucket-configuration LocationConstraint=${region}
Deploy VPC and DHCP server
$ export vpc=$(aws ec2 create-vpc --cidr-block ${cidrvpc} | jq -r '.') $ if [ $region == "us-east-1" ]; then export vpcdhcpopts_dnsdomain="ec2.internal" else export vpcdhcpopts_dnsdomain="${region}.compute.internal" fi $ export vpcdhcpopts=$(aws ec2 create-dhcp-options \ --dhcp-configuration " \ [ \ { \"Key\": \"domain-name\", \"Values\": [ \"${vpcdhcpopts_dnsdomain}\" ] }, \ { \"Key\": \"domain-name-servers\", \"Values\": [ \"AmazonProvidedDNS\" ] } \ ]") $ aws ec2 modify-vpc-attribute \ --enable-dns-hostnames \ --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId') $ aws ec2 modify-vpc-attribute \ --enable-dns-support \ --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId') $ aws ec2 associate-dhcp-options \ --dhcp-options-id $(echo ${vpcdhcpopts} | jq -r '.DhcpOptions.DhcpOptionsId') \ --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId')
Deploy IGW and attach to VPC
$ export igw=$(aws ec2 create-internet-gateway) $ aws ec2 attach-internet-gateway \ --internet-gateway-id $(echo ${igw} | jq -r '.InternetGateway.InternetGatewayId') \ --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId')
Deploy subnets
$ for i in $(seq 1 3); do export subnet${i}_public="$(aws ec2 create-subnet \ --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId') \ --cidr-block ${cidrsubnets_public[$(expr $i - 1 )]} \ --availability-zone ${az[$(expr $i - 1)]} )" done $ for i in $(seq 1 3); do export subnet${i}_private="$(aws ec2 create-subnet \ --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId') \ --cidr-block ${cidrsubnets_private[$(expr $i - 1 )]} \ --availability-zone ${az[$(expr $i - 1)]} )" done
Deploy EIPs
$ for i in $(seq 0 3); do export eip${i}="$(aws ec2 allocate-address --domain vpc)" done
Deploy NatGW’s
$ for i in $(seq 1 3); do j="eip${i}" k="subnet${i}_public" export natgw${i}="$(aws ec2 create-nat-gateway \ --subnet-id $(echo ${!k} | jq -r '.Subnet.SubnetId') \ --allocation-id $(echo ${!j} | jq -r '.AllocationId') \ )" done
Deploy RouteTables and routes
$ export routetable0=$(aws ec2 create-route-table \ --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId') ) $ aws ec2 create-route \ --route-table-id $(echo ${routetable0} | jq -r '.RouteTable.RouteTableId') \ --destination-cidr-block 0.0.0.0/0 \ --nat-gateway-id $(echo ${igw} | jq -r '.InternetGateway.InternetGatewayId') \ > /dev/null 2>&1 $ for i in $(seq 1 3); do j="subnet${i}_public" aws ec2 associate-route-table \ --route-table-id $(echo ${routetable0} | jq -r '.RouteTable.RouteTableId') \ --subnet-id $(echo ${!j} | jq -r '.Subnet.SubnetId') done $ for i in $(seq 1 3); do export routetable${i}="$(aws ec2 create-route-table \ --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId') \ )" done $ for i in $(seq 1 3); do j="routetable${i}" k="natgw${i}" aws ec2 create-route \ --route-table-id $(echo ${!j} | jq -r '.RouteTable.RouteTableId') \ --destination-cidr-block 0.0.0.0/0 \ --nat-gateway-id $(echo ${!k} | jq -r '.NatGateway.NatGatewayId') \ > /dev/null 2>&1 done $ for i in $(seq 1 3); do j="routetable${i}" k="subnet${i}_private" aws ec2 associate-route-table \ --route-table-id $(echo ${!j} | jq -r '.RouteTable.RouteTableId') \ --subnet-id $(echo ${!k} | jq -r '.Subnet.SubnetId') done
Deploy SecurityGroups and rules
$ for i in Bastion infra master node; do export awssg_$(echo ${i} | tr A-Z a-z)="$(aws ec2 create-security-group \ --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId') \ --group-name ${i} \ --description ${i})" done $ aws ec2 authorize-security-group-ingress \ --group-id $(echo ${awssg_bastion} | jq -r '.GroupId') \ --ip-permissions '[ {"IpProtocol": "icmp", "FromPort": 8, "ToPort": -1, "IpRanges": [{"CidrIp": "0.0.0.0/0"}]}, {"IpProtocol": "tcp", "FromPort": 22, "ToPort": 22, "IpRanges": [{"CidrIp": "0.0.0.0/0"}]} ]' $ aws ec2 authorize-security-group-ingress \ --group-id $(echo ${awssg_infra} | jq -r '.GroupId') \ --ip-permissions '[ {"IpProtocol": "tcp", "FromPort": 80, "ToPort": 80, "IpRanges": [{"CidrIp": "0.0.0.0/0"}]}, {"IpProtocol": "tcp", "FromPort": 443, "ToPort": 443, "IpRanges": [{"CidrIp": "0.0.0.0/0"}]} ]' $ aws ec2 authorize-security-group-ingress \ --group-id $(echo ${awssg_master} | jq -r '.GroupId') \ --ip-permissions '[ {"IpProtocol": "tcp", "FromPort": 443, "ToPort": 443, "IpRanges": [{"CidrIp": "0.0.0.0/0"}]} ]' $ for i in 2379-2380; do aws ec2 authorize-security-group-ingress \ --group-id $(echo ${awssg_master} | jq -r '.GroupId') \ --protocol tcp \ --port $i \ --source-group $(echo ${awssg_master} | jq -r '.GroupId') done $ for i in 2379-2380; do aws ec2 authorize-security-group-ingress \ --group-id $(echo ${awssg_master} | jq -r '.GroupId') \ --protocol tcp \ --port $i \ --source-group $(echo ${awssg_node} | jq -r '.GroupId') done $ aws ec2 authorize-security-group-ingress \ --group-id $(echo ${awssg_node} | jq -r '.GroupId') \ --ip-permissions '[ {"IpProtocol": "icmp", "FromPort": 8, "ToPort": -1, "IpRanges": [{"CidrIp": "0.0.0.0/0"}]} ]' $ aws ec2 authorize-security-group-ingress \ --group-id $(echo ${awssg_node} | jq -r '.GroupId') \ --protocol tcp \ --port 22 \ --source-group $(echo ${awssg_bastion} | jq -r '.GroupId') $ for i in 53 2049 8053 10250; do aws ec2 authorize-security-group-ingress \ --group-id $(echo ${awssg_node} | jq -r '.GroupId') \ --protocol tcp \ --port $i \ --source-group $(echo ${awssg_node} | jq -r '.GroupId') done $ for i in 53 4789 8053; do aws ec2 authorize-security-group-ingress \ --group-id $(echo ${awssg_node} | jq -r '.GroupId') \ --protocol udp \ --port $i \ --source-group $(echo ${awssg_node} | jq -r '.GroupId') done
Deploy ELB’s
$ export elb_masterext=$(aws elb create-load-balancer \ --load-balancer-name ${clusterid}-public-master \ --subnets \ $(echo ${subnet1_public} | jq -r '.Subnet.SubnetId') \ $(echo ${subnet2_public} | jq -r '.Subnet.SubnetId') \ $(echo ${subnet3_public} | jq -r '.Subnet.SubnetId') \ --listener Protocol=TCP,LoadBalancerPort=443,InstanceProtocol=TCP,InstancePort=443 \ --security-groups $(echo ${awssg_master} | jq -r '.GroupId') \ --scheme internet-facing \ --tags Key=name,Value=${clusterid}-public-master Key=Clusterid,Value=${clusterid}) $ aws elb modify-load-balancer-attributes \ --load-balancer-name ${clusterid}-public-master \ --load-balancer-attributes "{ \"CrossZoneLoadBalancing\":{\"Enabled\":true}, \"ConnectionDraining\":{\"Enabled\":false} }" $ aws elb configure-health-check \ --load-balancer-name ${clusterid}-public-master \ --health-check Target=HTTPS:443/api,HealthyThreshold=3,Interval=5,Timeout=2,UnhealthyThreshold=2 $ export elb_masterint=$(aws elb create-load-balancer \ --load-balancer-name ${clusterid}-private-master \ --subnets \ $(echo ${subnet1_private} | jq -r '.Subnet.SubnetId') \ $(echo ${subnet2_private} | jq -r '.Subnet.SubnetId') \ $(echo ${subnet3_private} | jq -r '.Subnet.SubnetId') \ --listener Protocol=TCP,LoadBalancerPort=443,InstanceProtocol=TCP,InstancePort=443 \ --security-groups $(echo ${awssg_master} | jq -r '.GroupId') \ --scheme internal \ --tags Key=name,Value=${clusterid}-private-master Key=Clusterid,Value=${clusterid}) $ aws elb modify-load-balancer-attributes \ --load-balancer-name ${clusterid}-private-master \ --load-balancer-attributes "{ \"CrossZoneLoadBalancing\":{\"Enabled\":true}, \"ConnectionDraining\":{\"Enabled\":false} }" $ aws elb configure-health-check \ --load-balancer-name ${clusterid}-private-master \ --health-check Target=HTTPS:443/api,HealthyThreshold=3,Interval=5,Timeout=2,UnhealthyThreshold=2 $ export elb_infraext=$(aws elb create-load-balancer \ --load-balancer-name ${clusterid}-public-infra \ --subnets \ $(echo ${subnet1_public} | jq -r '.Subnet.SubnetId') \ $(echo ${subnet2_public} | jq -r '.Subnet.SubnetId') \ $(echo ${subnet3_public} | jq -r '.Subnet.SubnetId') \ --listener \ Protocol=TCP,LoadBalancerPort=80,InstanceProtocol=TCP,InstancePort=80 \ Protocol=TCP,LoadBalancerPort=443,InstanceProtocol=TCP,InstancePort=443 \ --security-groups $(echo ${awssg_infra} | jq -r '.GroupId') \ --scheme internet-facing \ --tags Key=name,Value=${clusterid}-public-infra Key=Clusterid,Value=${clusterid}) $ aws elb modify-load-balancer-attributes \ --load-balancer-name ${clusterid}-public-infra \ --load-balancer-attributes "{ \"CrossZoneLoadBalancing\":{\"Enabled\":true}, \"ConnectionDraining\":{\"Enabled\":false} }" $ aws elb configure-health-check \ --load-balancer-name ${clusterid}-public-infra \ --health-check Target=TCP:443,HealthyThreshold=2,Interval=5,Timeout=2,UnhealthyThreshold=2
Deploy Route53 zones and resources
$ export route53_extzone=$(aws route53 create-hosted-zone \
--caller-reference $(date +%s) \
--name ${clusterid}.${dns_domain} \
--hosted-zone-config "PrivateZone=False")
$ export route53_intzone=$(aws route53 create-hosted-zone \
--caller-reference $(date +%s) \
--name ${clusterid}.${dns_domain} \
--vpc "VPCRegion=${region},VPCId=$(echo ${vpc} | jq -r '.Vpc.VpcId')" \
--hosted-zone-config "PrivateZone=True")
$ aws route53 change-resource-record-sets \
--hosted-zone-id $(echo ${route53_extzone} | jq -r '.HostedZone.Id' | sed 's/\/hostedzone\///g') \
--change-batch "\
{ \
\"Changes\": [ \
{ \
\"Action\": \"CREATE\", \
\"ResourceRecordSet\": { \
\"Name\": \"master.${clusterid}.${dns_domain}\", \
\"Type\": \"CNAME\", \
\"TTL\": 300, \
\"ResourceRecords\": [ \
{ \"Value\": \"$(echo ${elb_masterext} | jq -r '.DNSName')\" } \
] \
} \
} \
] \
}"
$ aws route53 change-resource-record-sets \
--hosted-zone-id $(echo ${route53_intzone} | jq -r '.HostedZone.Id' | sed 's/\/hostedzone\///g') \
--change-batch "\
{ \
\"Changes\": [ \
{ \
\"Action\": \"CREATE\", \
\"ResourceRecordSet\": { \
\"Name\": \"master.${clusterid}.${dns_domain}\", \
\"Type\": \"CNAME\", \
\"TTL\": 300, \
\"ResourceRecords\": [ \
{ \"Value\": \"$(echo ${elb_masterint} | jq -r '.DNSName')\" } \
] \
} \
} \
] \
}"
$ aws route53 change-resource-record-sets \
--hosted-zone-id $(echo ${route53_extzone} | jq -r '.HostedZone.Id' | sed 's/\/hostedzone\///g') \
--change-batch "\
{ \
\"Changes\": [ \
{ \
\"Action\": \"CREATE\", \
\"ResourceRecordSet\": { \
\"Name\": \".apps.${clusterid}.${dns_domain}\", \ \"Type\": \"CNAME\", \ \"TTL\": 300, \ \"ResourceRecords\": [ \ { \"Value\": \"$(echo ${elb_infraext} | jq -r '.DNSName')\" } \ ] \ } \ } \ ] \ }" $ aws route53 change-resource-record-sets \ --hosted-zone-id $(echo ${route53_intzone} | jq -r '.HostedZone.Id' | sed 's/\/hostedzone\///g') \ --change-batch "\ { \ \"Changes\": [ \ { \ \"Action\": \"CREATE\", \ \"ResourceRecordSet\": { \ \"Name\": \".apps.${clusterid}.${dns_domain}\", \
\"Type\": \"CNAME\", \
\"TTL\": 300, \
\"ResourceRecords\": [ \
{ \"Value\": \"$(echo ${elb_infraext} | jq -r '.DNSName')\" } \
] \
} \
} \
] \
}"
Create EC2 user-data script
$ cat << EOF > /tmp/ec2_userdata.sh #!/bin/bash if [ "\$#" -ne 2 ]; then exit 2; fi printf '%s\n' "#cloud-config" printf '%s' "cloud_config_modules:" if [ "\$1" == 'bastion' ]; then printf '\n%s\n\n' "- package-update-upgrade-install" else printf '\n%s\n\n' "- package-update-upgrade-install - disk_setup - mounts - cc_write_files" fi printf '%s' "packages:" if [ "\$1" == 'bastion' ]; then printf '\n%s\n' "- nmap-ncat" else printf '\n%s\n' "- lvm2" fi if [ "\$1" != 'bastion' ]; then printf '\n%s' 'write_files: - content: | STORAGE_DRIVER=overlay2 DEVS=/dev/' if [[ "\$2" =~ (c5|c5d|i3.metal|m5) ]]; then printf '%s' 'nvme1n1' else printf '%s' 'xvdb' fi printf '\n%s\n' ' VG=dockervg CONTAINER_ROOT_LV_NAME=dockerlv CONTAINER_ROOT_LV_MOUNT_PATH=/var/lib/docker CONTAINER_ROOT_LV_SIZE=100%FREE path: "/etc/sysconfig/docker-storage-setup" permissions: "0644" owner: "root"' printf '\n%s' 'fs_setup:' printf '\n%s' '- label: ocp_emptydir filesystem: xfs device: /dev/' if [[ "\$2" =~ (c5|c5d|i3.metal|m5) ]]; then printf '%s\n' 'nvme2n1' else printf '%s\n' 'xvdc' fi printf '%s' ' partition: auto' if [ "\$1" == 'master' ]; then printf '\n%s' '- label: etcd filesystem: xfs device: /dev/' if [[ "\$2" =~ (c5|c5d|i3.metal|m5) ]]; then printf '%s\n' 'nvme3n1' else printf '%s\n' 'xvdd' fi printf '%s' ' partition: auto' fi printf '\n' printf '\n%s' 'mounts:' printf '\n%s' '- [ "LABEL=ocp_emptydir", "/var/lib/origin/openshift.local.volumes", xfs, "defaults,gquota" ]' if [ "\$1" == 'master' ]; then printf '\n%s' '- [ "LABEL=etcd", "/var/lib/etcd", xfs, "defaults,gquota" ]' fi printf '\n' fi EOF $ chmod u+x /tmp/ec2_userdata.sh
Deploy the bastion EC2, sleep for 15 sec while AWS instantiates the EC2, and associate an EIP with the bastion for direct Internet access.
$ export ec2_bastion=$(aws ec2 run-instances \ --image-id ${ec2ami[1]} \ --count 1 \ --instance-type ${ec2_type_bastion} \ --key-name ${clusterid}.${dns_domain} \ --security-group-ids $(echo ${awssg_bastion} | jq -r '.GroupId') \ --subnet-id $(echo ${subnet1_public} | jq -r '.Subnet.SubnetId') \ --associate-public-ip-address \ --block-device-mappings "DeviceName=/dev/sda1,Ebs={DeleteOnTermination=False,VolumeSize=25}" \ --user-data "$(/tmp/ec2_userdata.sh bastion ${ec2_type_bastion})" \ --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=bastion},{Key=Clusterid,Value=${clusterid}}]" \ ) $ sleep 15 $ aws ec2 associate-address \ --allocation-id $(echo ${eip0} | jq -r '.AllocationId') \ --instance-id $(echo ${ec2_bastion} | jq -r '.Instances[].InstanceId')
Create the master, infra and node EC2s along with EBS volumes
$ for i in $(seq 1 3); do j="subnet${i}_private" export ec2_master${i}="$(aws ec2 run-instances \ --image-id ${ec2ami[1]} \ --count 1 \ --instance-type ${ec2_type_master} \ --key-name ${clusterid}.${dns_domain} \ --security-group-ids $(echo ${awssg_master} | jq -r '.GroupId') $(echo ${awssg_node} | jq -r '.GroupId') \ --subnet-id $(echo ${!j} | jq -r '.Subnet.SubnetId') \ --block-device-mappings \ "DeviceName=/dev/sda1,Ebs={DeleteOnTermination=False,VolumeSize=100}" \ "DeviceName=/dev/xvdb,Ebs={DeleteOnTermination=False,VolumeSize=100}" \ "DeviceName=/dev/xvdc,Ebs={DeleteOnTermination=False,VolumeSize=100}" \ "DeviceName=/dev/xvdd,Ebs={DeleteOnTermination=False,VolumeSize=100}" \ --user-data "$(/tmp/ec2_userdata.sh master ${ec2_type_master})" \ --tag-specifications "ResourceType=instance,Tags=[ \ {Key=Name,Value=master${i}}, \ {Key=Clusterid,Value=${clusterid}}, \ {Key=ami,Value=${ec2ami}}, \ {Key=kubernetes.io/cluster/${clusterid},Value=${clusterid}}]" )" done $ for i in $(seq 1 3); do j="subnet${i}_private" export ec2_infra${i}="$(aws ec2 run-instances \ --image-id ${ec2ami[1]} \ --count 1 \ --instance-type ${ec2_type_infra} \ --key-name ${clusterid}.${dns_domain} \ --security-group-ids $(echo ${awssg_infra} | jq -r '.GroupId') $(echo ${awssg_node} | jq -r '.GroupId') \ --subnet-id $(echo ${!j} | jq -r '.Subnet.SubnetId') \ --block-device-mappings \ "DeviceName=/dev/sda1,Ebs={DeleteOnTermination=False,VolumeSize=100}" \ "DeviceName=/dev/xvdb,Ebs={DeleteOnTermination=False,VolumeSize=100}" \ "DeviceName=/dev/xvdc,Ebs={DeleteOnTermination=False,VolumeSize=100}" \ "DeviceName=/dev/xvdd,Ebs={DeleteOnTermination=False,VolumeSize=100}" \ --user-data "$(/tmp/ec2_userdata.sh infra ${ec2_type_infra})" \ --tag-specifications "ResourceType=instance,Tags=[ \ {Key=Name,Value=infra${i}}, \ {Key=Clusterid,Value=${clusterid}}, \ {Key=ami,Value=${ec2ami}}, \ {Key=kubernetes.io/cluster/${clusterid},Value=${clusterid}}]" )" done $ for i in $(seq 1 3); do j="subnet${i}_private" export ec2_node${i}="$(aws ec2 run-instances \ --image-id ${ec2ami[1]} \ --count 1 \ --instance-type ${ec2_type_node} \ --key-name ${clusterid}.${dns_domain} \ --security-group-ids $(echo ${awssg_node} | jq -r '.GroupId') \ --subnet-id $(echo ${!j} | jq -r '.Subnet.SubnetId') \ --block-device-mappings \ "DeviceName=/dev/sda1,Ebs={DeleteOnTermination=False,VolumeSize=100}" \ "DeviceName=/dev/xvdb,Ebs={DeleteOnTermination=False,VolumeSize=100}" \ "DeviceName=/dev/xvdc,Ebs={DeleteOnTermination=False,VolumeSize=100}" \ --user-data "$(/tmp/ec2_userdata.sh node ${ec2_type_node})" \ --tag-specifications "ResourceType=instance,Tags=[ \ {Key=Name,Value=node${i}}, \ {Key=Clusterid,Value=${clusterid}}, \ {Key=ami,Value=${ec2ami}}, \ {Key=kubernetes.io/cluster/${clusterid},Value=${clusterid}}]" )" done $ rm -rf /tmp/ec2_userdata*
Register EC2s to ELB’s
$ export elb_masterextreg=$(aws elb register-instances-with-load-balancer \ --load-balancer-name ${clusterid}-public-master \ --instances \ $(echo ${ec2_master1} | jq -r '.Instances[].InstanceId') \ $(echo ${ec2_master2} | jq -r '.Instances[].InstanceId') \ $(echo ${ec2_master3} | jq -r '.Instances[].InstanceId') \ ) $ export elb_masterintreg=$(aws elb register-instances-with-load-balancer \ --load-balancer-name ${clusterid}-private-master \ --instances \ $(echo ${ec2_master1} | jq -r '.Instances[].InstanceId') \ $(echo ${ec2_master2} | jq -r '.Instances[].InstanceId') \ $(echo ${ec2_master3} | jq -r '.Instances[].InstanceId') \ ) $ export elb_infrareg=$(aws elb register-instances-with-load-balancer \ --load-balancer-name ${clusterid}-public-infra \ --instances \ $(echo ${ec2_infra1} | jq -r '.Instances[].InstanceId') \ $(echo ${ec2_infra2} | jq -r '.Instances[].InstanceId') \ $(echo ${ec2_infra3} | jq -r '.Instances[].InstanceId') \ )
Create tags on AWS components
aws ec2 create-tags --resources $(echo $vpc | jq -r ".Vpc.VpcId") --tags Key=Name,Value=${clusterid}; \ aws ec2 create-tags --resources $(echo ${eip0} | jq -r '.AllocationId') --tags Key=Name,Value=bastion; \ aws ec2 create-tags --resources $(echo ${eip1} | jq -r '.AllocationId') --tags Key=Name,Value=${az[0]}; \ aws ec2 create-tags --resources $(echo ${eip2} | jq -r '.AllocationId') --tags Key=Name,Value=${az[1]}; \ aws ec2 create-tags --resources $(echo ${eip3} | jq -r '.AllocationId') --tags Key=Name,Value=${az[2]}; \ aws ec2 create-tags --resources $(echo ${natgw1} | jq -r '.NatGateway.NatGatewayId') --tags Key=Name,Value=${az[0]}; \ aws ec2 create-tags --resources $(echo ${natgw2} | jq -r '.NatGateway.NatGatewayId') --tags Key=Name,Value=${az[1]}; \ aws ec2 create-tags --resources $(echo ${natgw3} | jq -r '.NatGateway.NatGatewayId') --tags Key=Name,Value=${az[2]}; \ aws ec2 create-tags --resources $(echo ${routetable0} | jq -r '.RouteTable.RouteTableId') --tags Key=Name,Value=routing; \ aws ec2 create-tags --resources $(echo ${routetable1} | jq -r '.RouteTable.RouteTableId') --tags Key=Name,Value=${az[0]}; \ aws ec2 create-tags --resources $(echo ${routetable2} | jq -r '.RouteTable.RouteTableId') --tags Key=Name,Value=${az[1]}; \ aws ec2 create-tags --resources $(echo ${routetable3} | jq -r '.RouteTable.RouteTableId') --tags Key=Name,Value=${az[2]}; \ aws ec2 create-tags --resources $(echo ${awssg_bastion} | jq -r '.GroupId') --tags Key=Name,Value=Bastion; \ aws ec2 create-tags --resources $(echo ${awssg_bastion} | jq -r '.GroupId') --tags Key=clusterid,Value=${clusterid}; \ aws ec2 create-tags --resources $(echo ${awssg_master} | jq -r '.GroupId') --tags Key=Name,Value=Master; \ aws ec2 create-tags --resources $(echo ${awssg_master} | jq -r '.GroupId') --tags Key=clusterid,Value=${clusterid}; \ aws ec2 create-tags --resources $(echo ${awssg_infra} | jq -r '.GroupId') --tags Key=Name,Value=Infra; \ aws ec2 create-tags --resources $(echo ${awssg_infra} | jq -r '.GroupId') --tags Key=clusterid,Value=${clusterid}; \ aws ec2 create-tags --resources $(echo ${awssg_node} | jq -r '.GroupId') --tags Key=Name,Value=Node; \ aws ec2 create-tags --resources $(echo ${awssg_node} | jq -r '.GroupId') --tags Key=clusterid,Value=${clusterid}
Create configuration files. These files are used to assist with installing
$ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain} #<!-- BEGIN OUTPUT --> Host bastion HostName $(echo ${eip0} | jq -r '.PublicIp') User ec2-user StrictHostKeyChecking no ProxyCommand none CheckHostIP no ForwardAgent yes ServerAliveInterval 15 TCPKeepAlive yes ControlMaster auto ControlPath ~/.ssh/mux-%r@%h:%p ControlPersist 15m ServerAliveInterval 30 IdentityFile ~/.ssh/${clusterid}.${dns_domain} Host *.compute-1.amazonaws.com ProxyCommand ssh -w 300 -W %h:%p bastion user ec2-user StrictHostKeyChecking no CheckHostIP no ServerAliveInterval 30 IdentityFile ~/.ssh/${clusterid}.${dns_domain} Host *.ec2.internal ProxyCommand ssh -w 300 -W %h:%p bastion user ec2-user StrictHostKeyChecking no CheckHostIP no ServerAliveInterval 30 IdentityFile ~/.ssh/${clusterid}.${dns_domain} #<!-- END OUTPUT --> EOF $ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain}-domaindelegation #<!-- BEGIN OUTPUT --> $(echo ${route53_extzone} | jq -r '.DelegationSet.NameServers[]') #<!-- END OUTPUT --> EOF $ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain}-cpkuser_access_key #<!-- BEGIN OUTPUT --> openshift_cloudprovider_aws_access_key=$(echo ${iamuser_accesskey} | jq -r '.AccessKey.AccessKeyId') openshift_cloudprovider_aws_secret_key=$(echo ${iamuser_accesskey} | jq -r '.AccessKey.SecretAccessKey') #<!-- END OUTPUT --> EOF $ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain}-cpk #<!-- BEGIN OUTPUT --> openshift_cloudprovider_kind=aws openshift_clusterid=${clusterid} EOF cat ~/.ssh/config-${clusterid}.${dns_domain}-cpkuser_access_key | \ grep -v 'OUTPUT -->' >> \ ~/.ssh/config-${clusterid}.${dns_domain}-cpk cat << EOF >> ~/.ssh/config-${clusterid}.${dns_domain}-cpk #<!-- END OUTPUT --> EOF $ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain}-s3user_access_key #<!-- BEGIN OUTPUT --> openshift_hosted_registry_storage_s3_accesskey=$(echo ${s3user_accesskey} | jq -r '.AccessKey.AccessKeyId') openshift_hosted_registry_storage_s3_secretkey=$(echo ${s3user_accesskey} | jq -r '.AccessKey.SecretAccessKey') #<!-- END OUTPUT --> EOF $ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain}-s3 #<!-- BEGIN OUTPUT --> openshift_hosted_manage_registry=true openshift_hosted_registry_storage_kind=object openshift_hosted_registry_storage_provider=s3 EOF cat ~/.ssh/config-${clusterid}.${dns_domain}-s3user_access_key | \ grep -v 'OUTPUT -->' >> \ ~/.ssh/config-${clusterid}.${dns_domain}-s3 cat << EOF >> ~/.ssh/config-${clusterid}.${dns_domain}-s3 openshift_hosted_registry_storage_s3_bucket=${clusterid}.${dns_domain}-registry openshift_hosted_registry_storage_s3_region=${region} openshift_hosted_registry_storage_s3_chunksize=26214400 openshift_hosted_registry_storage_s3_rootdirectory=/registry openshift_hosted_registry_pullthrough=true openshift_hosted_registry_acceptschema2=true openshift_hosted_registry_enforcequota=true openshift_hosted_registry_replicas=3 openshift_hosted_registry_selector='region=infra' #<!-- END OUTPUT --> EOF $ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain}-urls #<!-- BEGIN OUTPUT --> openshift_master_default_subdomain=apps.${clusterid}.${dns_domain} openshift_master_cluster_hostname=master.${clusterid}.${dns_domain} openshift_master_cluster_public_hostname=master.${clusterid}.${dns_domain} #<!-- END OUTPUT --> EOF $ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain}-hosts [masters] $(echo ${ec2_master1} | jq -r '.Instances[].PrivateDnsName') openshift_node_labels="{'region': 'master'}" $(echo ${ec2_master2} | jq -r '.Instances[].PrivateDnsName') openshift_node_labels="{'region': 'master'}" $(echo ${ec2_master3} | jq -r '.Instances[].PrivateDnsName') openshift_node_labels="{'region': 'master'}" [etcd] [etcd:children] masters [nodes] $(echo ${ec2_node1} | jq -r '.Instances[].PrivateDnsName') openshift_node_labels="{'region': 'apps'}" $(echo ${ec2_node2} | jq -r '.Instances[].PrivateDnsName') openshift_node_labels="{'region': 'apps'}" $(echo ${ec2_node3} | jq -r '.Instances[].PrivateDnsName') openshift_node_labels="{'region': 'apps'}" $(echo ${ec2_infra1} | jq -r '.Instances[].PrivateDnsName') openshift_node_labels="{'region': 'infra', 'zone': 'default'}" $(echo ${ec2_infra2} | jq -r '.Instances[].PrivateDnsName') openshift_node_labels="{'region': 'infra', 'zone': 'default'}" $(echo ${ec2_infra3} | jq -r '.Instances[].PrivateDnsName') openshift_node_labels="{'region': 'infra', 'zone': 'default'}" [nodes:children] masters EOF
2.11.2. Ansible
GitHub OpenShift openshift-ansible-contrib repository contains an Ansible playbook that provides a friendly and repeatable deployment experience.
Clone openshift-ansible-contrib repository then enter the reference architecture 3.9 directory.
$ git clone https://github.com/openshift/openshift-ansible-contrib.git $ cd openshift-ansible-contrib/reference-architecture/3.9
Create a simple Ansible inventory
$ sudo vi /etc/ansible/hosts [local] 127.0.0.1 [local:vars] ansible_connection=local ansible_become=False
Review playbooks/vars/main.yaml
now and ensure values fit requirements.
When values are satisfactory execute deploy_aws.yaml
play to deploy AWS infrastructure.
$ ansible-playbook playbooks/deploy_aws.yaml
Troubleshooting the deploy_aws.yaml
play is simple. It is self repairing. If any errors are encountered simply rerun play.
2.12. Create AWS Infrastructure for Red Hat OpenShift Container Platform CNS (Optional)
2.12.1. CLI
Set environment variables to be sourced by other commands. These values can be modified to fit any environment.
$ export ec2_type_cns="m5.2xlarge"
Deploy SecurityGroup and rules
$ export awssg_cns=$(aws ec2 create-security-group \ --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId') \ --group-name cns \ --description "cns") $ for i in 111 2222 3260 24007-24008 24010 49152-49664; do aws ec2 authorize-security-group-ingress \ --group-id $(echo ${awssg_cns} | jq -r '.GroupId') \ --protocol tcp \ --port $i \ --source-group $(echo ${awssg_cns} | jq -r '.GroupId') done $ for i in 3260 24007-24008 24010 49152-49664; do aws ec2 authorize-security-group-ingress \ --group-id $(echo ${awssg_cns} | jq -r '.GroupId') \ --protocol tcp \ --port $i \ --source-group $(echo ${awssg_node} | jq -r '.GroupId') done $ aws ec2 authorize-security-group-ingress \ --group-id $(echo ${awssg_cns} | jq -r '.GroupId') \ --protocol udp \ --port 111 \ --source-group $(echo ${awssg_cns} | jq -r '.GroupId')
Create node EC2 instances along with EBS volumes
$ for i in 1 2 3; do j="subnet${i}_private" export ec2_cns${i}="$(aws ec2 run-instances \ --image-id ${ec2ami[1]} \ --count 1 \ --instance-type ${ec2_type_cns} \ --key-name ${clusterid}.${dns_domain} \ --security-group-ids $(echo ${awssg_cns} | jq -r '.GroupId') $(echo ${awssg_node} | jq -r '.GroupId') \ --subnet-id $(echo ${!j} | jq -r '.Subnet.SubnetId') \ --block-device-mappings \ "DeviceName=/dev/sda1,Ebs={DeleteOnTermination=False,VolumeSize=100}" \ "DeviceName=/dev/xvdb,Ebs={DeleteOnTermination=False,VolumeSize=100}" \ "DeviceName=/dev/xvdc,Ebs={DeleteOnTermination=False,VolumeSize=100}" \ "DeviceName=/dev/xvdd,Ebs={DeleteOnTermination=False,VolumeSize=100}" \ --user-data "$(/tmp/ec2_userdata.sh node ${ec2_type_node})" \ --tag-specifications "ResourceType=instance,Tags=[ \ {Key=Name,Value=cns${i}}, \ {Key=Clusterid,Value=${clusterid}}, \ {Key=ami,Value=${ec2ami}}, \ {Key=kubernetes.io/cluster/${clusterid},Value=${clusterid}}]" )" done
Create configuration files. These files are used to assist with installing Red Hat OpenShift Container Platform.
$ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain}-hostscns $(echo ${ec2_cns1} | jq -r '.Instances[].PrivateDnsName') openshift_schedulable=True $(echo ${ec2_cns2} | jq -r '.Instances[].PrivateDnsName') openshift_schedulable=True $(echo ${ec2_cns3} | jq -r '.Instances[].PrivateDnsName') openshift_schedulable=True EOF $ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain}-hostsgfs [glusterfs] $(echo ${ec2_cns1} | jq -r '.Instances[].PrivateDnsName') glusterfs_devices='[ "/dev/nvme3n1" ]' $(echo ${ec2_cns2} | jq -r '.Instances[].PrivateDnsName') glusterfs_devices='[ "/dev/nvme3n1" ]' $(echo ${ec2_cns3} | jq -r '.Instances[].PrivateDnsName') glusterfs_devices='[ "/dev/nvme3n1" ]' EOF
2.12.2. Ansible
Run deploy_aws_cns.yaml
play to deploy additional infrastructure for CNS.
$ ansible-playbook playbooks/deploy_aws_cns.yaml
deploy_aws_cns.yaml
playbook requires deploy_aws.yaml
to be previously run. Troubleshooting the deploy_aws_cns.yaml
play is simple. It is self repairing. If any errors are encountered simply rerun play.
2.13. Bastion Configuration
It is often preferrable to use a bastion host as an ssh proxy for security. Awscli and Ansible both output a ssh configuration file to enable direct ssh connections to Red Hat OpenShift Container Platform instances.
Perform the following procedure to set configuration:
$ mv ~/.ssh/config ~/.ssh/config-orig $ ln -s ~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN > ~/.ssh/config $ chmod 400 ~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN >
Perform the following procedure to ensure ssh-agent is running and local ssh key can be used to proxy connections to target EC2 instances:
$ if [ ! "$(env | grep SSH_AGENT_PID)" ] || [ ! "$(ps -ef | grep -v grep | grep ${SSH_AGENT_PID})" ]; then rm -rf ${SSH_AUTH_SOCK} 2> /dev/null unset SSH_AUTH_SOCK unset SSH_AGENT_PID pkill ssh-agent export sshagent=$(nohup ssh-agent &) export sshauthsock=$(echo ${sshagent} | awk -F'; ' {'print $1'}) export sshagentpid=$(echo ${sshagent} | awk -F'; ' {'print $3'}) export ${sshauthsock} export ${sshagentpid} for i in sshagent sshauthsock sshagentpid; do unset $i done fi $ export sshkey=($(cat ~/.ssh/< CLUSTERID >.< DNS_DOMAIN >.pub)) $ IFS=$'\n'; if [ ! $(ssh-add -L | grep ${sshkey[1]}) ]; then ssh-add ~/.ssh/< CLUSTERID >.< DNS_DOMAIN > fi $ unset IFS
Verify < CLUSTERID >.< DNS_DOMAIN > ssh key is added to ssh-agent
$ ssh-add -l $ ssh-add -L
2.14. OpenShift Preparations
Once the instances have been deployed and the ~/.ssh/config
file reflects the deployment the following steps should be performed to prepare for the installation of OpenShift.
2.14.1. Public domain delegation
To access Red Hat OpenShift Container Platform on AWS from the public Internet delegation for the subdomain must be setup using the following information.
DNS subdomain | Route53 public zone NS records |
---|---|
< CLUSTERID >.< DNS_DOMAIN > | See file ~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN >-domaindelegation for correct values |
WARN: DNS service provder domain delegation is out of scope of this guide. A customer will need to follow their providers best practise in delegation setup.
2.14.2. OpenShift Authentication
Red Hat OpenShift Container Platform provides the ability to use many different authentication platforms.
Detailed listing of other authentication providers are available at Configuring Authentication and User Agent.
For this reference architecture Google’s OpenID Connect Integration is used. When configuring the authentication, the following parameters must be added to the ansible inventory. An example is shown below.
openshift_master_identity_providers=[{'name': 'google', 'challenge': 'false', 'login': 'true', 'kind': 'GoogleIdentityProvider', 'mapping_method': 'claim', 'clientID': '246358064255-5ic2e4b1b9ipfa7hddfkhuf8s6eq2rfj.apps.googleusercontent.com', 'clientSecret': 'Za3PWZg7gQxM26HBljgBMBBF', 'hostedDomain': 'redhat.com'}]
2.14.3. Openshift-ansible Installer Inventory
This section provides an example inventory file required for an advanced installation of Red Hat OpenShift Container Platform.
The inventory file contains both variables and instances used for the configuration and deployment of Red Hat OpenShift Container Platform.
$ sudo vi /etc/ansible/hosts [OSEv3:children] masters etcd nodes [OSEv3:vars] debug_level=2 ansible_user=ec2-user ansible_become=yes openshift_deployment_type=openshift-enterprise openshift_release=v3.9 openshift_master_api_port=443 openshift_master_console_port=443 openshift_portal_net=172.30.0.0/16 os_sdn_network_plugin_name='redhat/openshift-ovs-networkpolicy' openshift_master_cluster_method=native container_runtime_docker_storage_setup_device=/dev/nvme1n1 openshift_node_local_quota_per_fsgroup=512Mi osm_use_cockpit=true openshift_hostname_check=false openshift_examples_modify_imagestreams=true oreg_url=registry.access.redhat.com/openshift3/ose-${component}:${version} openshift_hosted_router_selector='region=infra' openshift_hosted_router_replicas=3 # UPDATE TO CORRECT IDENTITY PROVIDER openshift_master_identity_providers=[{'name': 'google', 'challenge': 'false', 'login': 'true', 'kind': 'GoogleIdentityProvider', 'mapping_method': 'claim', 'clientID': '246358064255-5ic2e4b1b9ipfa7hddfkhuf8s6eq2rfj.apps.googleusercontent.com', 'clientSecret': 'Za3PWZg7gQxM26HBljgBMBBF', 'hostedDomain': 'redhat.com'}] # UPDATE USING VALUES FOUND IN awscli env vars or~/playbooks/var/main.yaml
# SEE ALSO FILE~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN >-urls
openshift_master_default_subdomain=apps.< CLUSTERID >.< DNS_DOMAIN > openshift_master_cluster_hostname=master.< CLUSTERID >.< DNS_DOMAIN > openshift_master_cluster_public_hostname=master.< CLUSTERID >.< DNS_DOMAIN > # UPDATE USING VALUES FOUND IN FILE~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN >-cpk
openshift_cloudprovider_kind=aws openshift_clusterid=refarch openshift_cloudprovider_aws_access_key=UPDATEACCESSKEY openshift_cloudprovider_aws_secret_key=UPDATESECRETKEY # UPDATE USING VALUES FOUND IN FILE~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN >-s3
openshift_hosted_manage_registry=true openshift_hosted_registry_storage_kind=object openshift_hosted_registry_storage_provider=s3 openshift_hosted_registry_storage_s3_accesskey=UPDATEACCESSKEY openshift_hosted_registry_storage_s3_secretkey=UPDATESECRETKEY openshift_hosted_registry_storage_s3_bucket=refarch-registry openshift_hosted_registry_storage_s3_region=REGION openshift_hosted_registry_storage_s3_chunksize=26214400 openshift_hosted_registry_storage_s3_rootdirectory=/registry openshift_hosted_registry_pullthrough=true openshift_hosted_registry_acceptschema2=true openshift_hosted_registry_enforcequota=true openshift_hosted_registry_replicas=3 openshift_hosted_registry_selector='region=infra' # Aggregated logging openshift_logging_install_logging=true openshift_logging_storage_kind=dynamic openshift_logging_storage_volume_size=25Gi openshift_logging_es_cluster_size=3 # Metrics openshift_metrics_install_metrics=true openshift_metrics_storage_kind=dynamic openshift_metrics_storage_volume_size=25Gi openshift_enable_service_catalog=true template_service_broker_install=true # UPDATE USING HOSTS FOUND IN FILE~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN >-hosts
[masters] ip-172-16-30-15.ec2.internal openshift_node_labels="{'region': 'master'}" ip-172-16-47-176.ec2.internal openshift_node_labels="{'region': 'master'}" ip-172-16-48-251.ec2.internal openshift_node_labels="{'region': 'master'}" [etcd] [etcd:children] masters # UPDATE USING HOSTS FOUND IN FILE~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN >-hosts
[nodes] ip-172-16-16-239.ec2.internal openshift_node_labels="{'region': 'apps'}" ip-172-16-37-179.ec2.internal openshift_node_labels="{'region': 'apps'}" ip-172-16-60-134.ec2.internal openshift_node_labels="{'region': 'apps'}" ip-172-16-17-209.ec2.internal openshift_node_labels="{'region': 'infra', 'zone': 'default'}" ip-172-16-46-136.ec2.internal openshift_node_labels="{'region': 'infra', 'zone': 'default'}" ip-172-16-56-149.ec2.internal openshift_node_labels="{'region': 'infra', 'zone': 'default'}" [nodes:children] masters
Advanced installations example configurations and options are provided in /usr/share/doc/openshift-ansible-docs-3.9.*/docs/example-inventories directory
.
2.14.4. CNS Inventory (Optional)
If CNS is used in the OpenShift installation specific variables must be set in the inventory.
$ sudo vi /etc/ansible/hosts # ADDglusterfs
to section [OSEv3:children] [OSEv3:children] masters etcd nodes glusterfs [OSEv3:vars] ....omitted... # CNS storage cluster openshift_storage_glusterfs_namespace=glusterfs openshift_storage_glusterfs_storageclass=true openshift_storage_glusterfs_storageclass_default=false openshift_storage_glusterfs_block_deploy=false openshift_storage_glusterfs_block_host_vol_create=false openshift_storage_glusterfs_block_host_vol_size=80 openshift_storage_glusterfs_block_storageclass=false openshift_storage_glusterfs_block_storageclass_default=false ....omitted... # ADD CNS HOSTS TO SECTION [nodes] # SEE INVENTORY IN FILE~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN >-hostscns
[nodes] ....omitted... ip-172-16-17-130.ec2.internal openshift_schedulable=True ip-172-16-40-219.ec2.internal openshift_schedulable=True ip-172-16-63-212.ec2.internal openshift_schedulable=True # ADD SECTION [glusterfs] # SEE INVENTORY IN FILE~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN >-hostsgfs
[glusterfs] ip-172-16-17-130.ec2.internal glusterfs_devices='[ "/dev/nvme3n1" ]' ip-172-16-40-219.ec2.internal glusterfs_devices='[ "/dev/nvme3n1" ]' ip-172-16-63-212.ec2.internal glusterfs_devices='[ "/dev/nvme3n1" ]'
2.14.5. EC2 instances
2.14.5.1. Test network connectivity
Ansible ping is used to test if the local environment can reach all instances. If there is a failure here then please check ssh config or network connectivity for the instance.
$ ansible nodes -b -m ping
2.14.5.2. Remove RHUI yum repos
Existing Red Hat Update Infrastructure yum repos must be disabled to avoid repository and rpm dependency conflicts with Red Hat OpenShift Container Platform repository and rpms.
$ ansible nodes -b -m command -a "yum-config-manager \ --disable 'rhui-REGION-client-config-server-7' \ --disable 'rhui-REGION-rhel-server-rh-common' \ --disable 'rhui-REGION-rhel-server-releases'"
2.14.5.3. Node Registration
Now that the inventory has been created the nodes must be subscribed using subscription-manager
.
The ad-hoc playbook below uses the redhat_subscription
module to register the instances. The first example uses the numeric pool value for the OpenShift subscription. The second uses an activation key and organization.
$ ansible nodes -b -m redhat_subscription -a \ "state=present username=USER password=PASSWORD pool_ids=NUMBERIC_POOLID"
OR
$ ansible nodes -b -m redhat_subscription -a \ "state=present activationkey=KEY org_id=ORGANIZATION"
2.14.5.4. Repository Setup
Once the instances are registered the proper repositories must be assigned to the instances to allow for packages for Red Hat OpenShift Container Platform to be installed.
$ ansible nodes -b -m shell -a \ 'subscription-manager repos --disable="*" \ --enable="rhel-7-server-rpms" \ --enable="rhel-7-server-extras-rpms" \ --enable="rhel-7-server-ose-3.9-rpms" \ --enable="rhel-7-fast-datapath-rpms"'
2.14.6. EmptyDir Storage
During the deployment of instances an extra volume is added to the instances for EmptyDir
storage. EmptyDir provides ephemeral (as opposed to persistent) storage for containers. The volume is used to help ensure that the /var volume does not get filled by containers using the storage.
The user-data script attached to EC2 instances automatically provisioned filesystem, added an entry to fstab, and mounted this volume.
2.14.7. etcd
Storage
During the deployment of the master instances an extra volume was added to the instances for etcd
storage. Having the separate disks specifically for etcd
ensures that all of the resources are available to the etcd
service such as i/o and total disk space.
The user-data script attached to EC2 instances automatically provisioned filesystem, added an entry to fstab, and mounted this volume.
2.14.8. Container Storage
The prerequisite playbook provided by the OpenShift Ansible RPMs configures container storage and installs any remaining packages for the installation.
$ ansible-playbook \ /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml