Chapter 2. Red Hat OpenShift Container Platform Prerequisites

2.1. Red Hat Cloud Access

AWS provides a RHEL based machine image to use as root volumes for EC2 virtual machines. AWS adds an additional per hour charge to EC2s running the image to cover the price of the license. This enables customers to pay one fee to AWS and not deal with multiple vendors. To other customers this may be a security risk as AWS has the duty of building the machine image and maintaining its lifecycle.

Red Hat provides a bring your own license program for Red Hat Enterprise Linux in AWS. Machine images are presented to participating customer for use as EC2 root volumes. Once signed up Red Hat machine images are then made available in the AMI Private Images inventory.

Red Hat Cloud Access program: https://www.redhat.com/en/technologies/cloud-computing/cloud-access

Note

This reference architecture expects customers to be Cloud Access participants

2.2. Execution environment

RHEL 7 is the only operating system supported by Red Hat OpenShift Container Platform installer therefore provider infrastructure deployment and installer must be run from one of the following locations running RHEL 7:

  • Local workstation / server / virtual machine / Bastion
  • Jenkins continuous delivery engine

This reference architecture focus’s on deploying and installing Red Hat OpenShift Container Platform from local workstation/server/virtual machine and bastion. Registering this host to Red Hat subscription manager will be a requirement to get access to Red Hat OpenShift Container Platform installer and related rpms. Jenkins is out of scope.

2.3. AWS IAM user

An IAM user with an admin policy and access key is required to interface with AWS API and deploy infrastructure using either AWS CLI or Boto AWS SDK.

Follow this procedure to create an IAM user https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html#id_users_create_console

Follow this procedure to create an access key for an IAM user https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html

2.4. EPEL and supporting rpms

epel repo contains supporting rpms to assist with tooling and Red Hat OpenShift Container Platform infrastructure deployment and installation.

Use the following command to enable the epel repo:

$ yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

python-pip is useful to install awscli, boto3/boto AWS python bindings.

$ yum install -y python-pip

jq simplifies data extraction from json output or files.

$ yum install -y jq

epel is no longer needed and can be removed.

$ yum erase epel-release

2.5. AWS Command Line Interface (awscli)

The awscli can be used to deploy all of the components associated with this Reference Architecture.

Detailed installation procedure is here: https://docs.aws.amazon.com/cli/latest/userguide/installing.html

This reference architecture supports the following minimum version of awscli:

$ aws --version
aws-cli/1.15.16 Python/2.7.5 Linux/3.10.0-862.el7.x86_64 botocore/1.8.50

2.6. Boto3 AWS SDK for Python

Boto3 is the AWS SDK for Python, which allows Python developers to write software that makes use of AWS services. Boto provides an easy to use, object-oriented API, and low-level direct service access.

Detailed Boto3 AWS python bindings installation procedure is here: http://boto3.readthedocs.io/en/latest/guide/quickstart.html#installation

Detailed legacy Boto AWS python bindings installation procedure is here: http://boto.cloudhackers.com/en/latest/getting_started.html

Note

If using Ansible to deploy AWS infrastructure installing boto3 AND legacy boto python bindings is mandatory as some Ansible modules still use the legacy boto AWS python bindings.

Ansible AWS tasks can experience random errors due the speed of execution and AWS API rate limiting. Follow this procedure to ensure Ansible tasks complete sucessfully:

$ cat << EOF > ~/.boto
[Boto]
debug = 0
num_retries = 10
EOF

This reference architecture supports the following minimum version of boto3:

$ pip freeze | grep boto3
boto3==1.5.24

This reference architecture supports the following minimum version of boto:

$ pip freeze | grep boto
boto==2.48.0

2.7. Set Up Client Credentials

Once the AWS CLI and Boto AWS SDK are installed, the credential needs to be set up.

Detailed IAM user authorization and authentication credentials procedure is here: https://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html

Shell environment variables can be an alternative to a static credential file.

$ export AWS_ACCESS_KEY_ID = <your aws_access_key_id here>

$ export AWS_SECRET_ACCESS_KEY = <your aws_secret_access_key here>

2.8. Client Testing

A successful awscli and credentials test will look similar to the following:

$ aws sts get-caller-identity

output:
{
    "Account": "123123123123",
    "UserId": "TH75ISMYR3F4RCHUS3R1D",
    "Arn": "arn:aws:iam::123123123123:user/refarchuser"
}

The following error indicates an issue with the AWS IAM user’s access key and local credentials:

$ aws sts get-caller-identity

output:
An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid.

A successful boto and credentials will look similar to the following:

$ cat << EOF | python
import boto3
print(boto3.client('sts').get_caller_identity()['Arn'])
EOF

output:
arn:aws:iam::123123123123:user/refarch

The following error indicates an issue with the AWS IAM user’s access key and local credentials:

output:
....
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid.

2.9. Git (optional)

Git is a fast, scalable, distributed revision control system with an unusually rich command set that provides both high-level operations and full access to internals.

Install git package

$ yum install -y git
Note

When using Ansible to deploy AWS infrastructure this step is mandatory as git is used to clone a source code repository.

2.10. Subscription Manager

Register instance with Red Hat Subscription Manager and activate yum repos

$ subscription-manager register

$ subscription-manager attach \
    --pool NUMBERIC_POOLID

$ subscription-manager repos \
    --disable="*" \
    --enable=rhel-7-server-rpms \
    --enable=rhel-7-server-extras-rpms \
    --enable=rhel-7-server-ansible-2.4-rpms \
    --enable=rhel-7-server-ose-3.9-rpms

$ yum -y install atomic-openshift-utils

2.11. Create AWS Infrastructure for Red Hat OpenShift Container Platform

Infrastructure creation can be executed anywhere provided the AWS API is network accessible and tooling requirements have been met. It is common for local workstations to be used for this step.

Note

When using a bastion as the installer execution environment be sure that all of the previous requirements get met. In some cases such as when an AWS EC2 is being used as a bastion there can be a chicken / egg issue where AWS network components and bastion instance to be used as the installer execution environment is not yet available.

2.11.1. CLI

The following awscli commands are provided to expose the complexity behind automation tools such Ansible.

Review the following environment variables now and ensure values fit requirements. When values are satisfactory execute the commands in terminal.

$ export clusterid="rhocp"
$ export dns_domain="example.com"
$ export region="us-east-1"
$ export cidrvpc="172.16.0.0/16"
$ export cidrsubnets_public=("172.16.0.0/24" "172.16.1.0/24" "172.16.2.0/24")
$ export cidrsubnets_private=("172.16.16.0/20" "172.16.32.0/20" "172.16.48.0/20")
$ export ec2_type_bastion="t2.medium"
$ export ec2_type_master="m5.2xlarge"
$ export ec2_type_infra="m5.2xlarge"
$ export ec2_type_node="m5.2xlarge"
$ export rhel_release="rhel-7.5"

Create a public private ssh keypair to be used with ssh-agent and ssh authentication on AWS EC2s.

$ if [ ! -f ${HOME}/.ssh/${clusterid}.${dns_domain} ]; then
  echo 'Enter ssh key password'
  read -r passphrase
  ssh-keygen -P ${passphrase} -o -t rsa -f ~/.ssh/${clusterid}.${dns_domain}
fi

$ export sshkey=($(cat ~/.ssh/${clusterid}.${dns_domain}.pub))

Gather available Availability Zones. The first 3 are used.

$ IFS=$' '; export az=($(aws ec2 describe-availability-zones \
    --filters "Name=region-name,Values=${region}" | \
    jq -r '.[][].ZoneName' | \
    head -3 | \
    tr '\n' ' ' | \
    sed -e "s/ $//g"));

$ unset IFS

Retrive Red Hat Cloud Access AMI

$ export ec2ami=($(aws ec2 describe-images \
    --region ${region} --owners 309956199498 | \
    jq -r '.Images[] | [.Name,.ImageId] | @csv' | \
    sed -e 's/,/ /g' | \
    sed -e 's/"//g' | \
    grep HVM_GA | \
    grep Access2-GP2 | \
    grep -i ${rhel_release} | \
    sort | \
    tail -1))

Deploy IAM users and sleep for 15 sec so that AWS can instantiate the users

$ export iamuser=$(aws iam create-user --user-name ${clusterid}.${dns_domain}-admin)

$ export s3user=$(aws iam create-user --user-name ${clusterid}.${dns_domain}-registry)

$ sleep 15

Create access key for IAM users

$ export iamuser_accesskey=$(aws iam create-access-key --user-name ${clusterid}.${dns_domain}-admin)

$ export s3user_accesskey=$(aws iam create-access-key --user-name ${clusterid}.${dns_domain}-registry)

Create and attach policies to IAM users

$ cat << EOF > ~/.iamuser_policy_cpk
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:DescribeVolume*",
                "ec2:CreateVolume",
                "ec2:CreateTags",
                "ec2:DescribeInstances",
                "ec2:AttachVolume",
                "ec2:DetachVolume",
                "ec2:DeleteVolume",
                "ec2:DescribeSubnets",
                "ec2:CreateSecurityGroup",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeRouteTables",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupIngress",
                "elasticloadbalancing:DescribeTags",
                "elasticloadbalancing:CreateLoadBalancerListeners",
                "elasticloadbalancing:ConfigureHealthCheck",
                "elasticloadbalancing:DeleteLoadBalancerListeners",
                "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:CreateLoadBalancer",
                "elasticloadbalancing:DeleteLoadBalancer",
                "elasticloadbalancing:ModifyLoadBalancerAttributes",
                "elasticloadbalancing:DescribeLoadBalancerAttributes"
            ],
            "Resource": "*",
            "Effect": "Allow",
            "Sid": "1"
        }
    ]
}
EOF

$ aws iam put-user-policy \
    --user-name ${clusterid}.${dns_domain}-admin \
    --policy-name Admin \
    --policy-document file://~/.iamuser_policy_cpk

$ cat << EOF > ~/.iamuser_policy_s3
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::${clusterid}.${dns_domain}-registry",
                "arn:aws:s3:::${clusterid}.${dns_domain}-registry/*"
            ],
            "Effect": "Allow",
            "Sid": "1"
        }
    ]
}
EOF

$ aws iam put-user-policy \
    --user-name ${clusterid}.${dns_domain}-registry \
    --policy-name S3 \
    --policy-document file://~/.iamuser_policy_s3

$ rm -rf ~/.iamuser_policy*

Deploy EC2 keypair

$ export keypair=$(aws ec2 import-key-pair \
    --key-name ${clusterid}.${dns_domain} \
    --public-key-material file://~/.ssh/${clusterid}.${dns_domain}.pub \
    )

Deploy S3 bucket and policy

$ export aws_s3bucket=$(aws s3api create-bucket \
    --bucket $(echo ${clusterid}.${dns_domain}-registry) \
    --region ${region} \
    )

$ aws s3api put-bucket-tagging \
    --bucket $(echo ${clusterid}.${dns_domain}-registry) \
    --tagging "TagSet=[{Key=Clusterid,Value=${clusterid}}]"

$ aws s3api put-bucket-policy \
    --bucket $(echo ${clusterid}.${dns_domain}-registry) \
    --policy "\
{ \
  \"Version\": \"2012-10-17\",
  \"Statement\": [ \
    { \
      \"Action\": \"s3:*\", \
      \"Effect\": \"Allow\", \
      \"Principal\": { \
        \"AWS\": \"$(echo ${s3user} | jq -r '.User.Arn')\" \
      }, \
      \"Resource\": \"arn:aws:s3:::$(echo ${aws_s3bucket} | jq -r '.Location' | sed -e 's/^\///g')\", \
      \"Sid\": \"1\" \
    } \
  ] \
}"
Warning

If region is a region other than us-east-1 then aws s3api create-bucket will require --create-bucket-configuration LocationConstraint=${region}

Deploy VPC and DHCP server

$ export vpc=$(aws ec2 create-vpc --cidr-block ${cidrvpc} | jq -r '.')

$ if [ $region == "us-east-1" ]; then
    export vpcdhcpopts_dnsdomain="ec2.internal"
else
    export vpcdhcpopts_dnsdomain="${region}.compute.internal"
fi

$ export vpcdhcpopts=$(aws ec2 create-dhcp-options \
    --dhcp-configuration " \
    [ \
      { \"Key\": \"domain-name\", \"Values\": [ \"${vpcdhcpopts_dnsdomain}\" ] }, \
      { \"Key\": \"domain-name-servers\", \"Values\": [ \"AmazonProvidedDNS\" ] } \
    ]")

$ aws ec2 modify-vpc-attribute \
    --enable-dns-hostnames \
    --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId')

$ aws ec2 modify-vpc-attribute \
    --enable-dns-support \
    --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId')

$ aws ec2 associate-dhcp-options \
    --dhcp-options-id $(echo ${vpcdhcpopts} | jq -r '.DhcpOptions.DhcpOptionsId') \
    --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId')

Deploy IGW and attach to VPC

$ export igw=$(aws ec2 create-internet-gateway)

$ aws ec2 attach-internet-gateway \
  --internet-gateway-id $(echo ${igw} | jq -r '.InternetGateway.InternetGatewayId') \
  --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId')

Deploy subnets

$ for i in $(seq 1 3); do
    export subnet${i}_public="$(aws ec2 create-subnet \
        --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId') \
        --cidr-block ${cidrsubnets_public[$(expr $i - 1 )]} \
        --availability-zone ${az[$(expr $i - 1)]}
    )"
done

$ for i in $(seq 1 3); do
    export subnet${i}_private="$(aws ec2 create-subnet \
        --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId') \
        --cidr-block ${cidrsubnets_private[$(expr $i - 1 )]} \
        --availability-zone ${az[$(expr $i - 1)]}
    )"
done

Deploy EIPs

$ for i in $(seq 0 3); do
    export eip${i}="$(aws ec2 allocate-address --domain vpc)"
done

Deploy NatGW’s

$ for i in $(seq 1 3); do
    j="eip${i}"
    k="subnet${i}_public"
    export natgw${i}="$(aws ec2 create-nat-gateway \
        --subnet-id $(echo ${!k} | jq -r '.Subnet.SubnetId') \
        --allocation-id $(echo ${!j} | jq -r '.AllocationId') \
    )"
done

Deploy RouteTables and routes

$ export routetable0=$(aws ec2 create-route-table \
    --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId')
    )

$ aws ec2 create-route \
    --route-table-id $(echo ${routetable0} | jq -r '.RouteTable.RouteTableId') \
    --destination-cidr-block 0.0.0.0/0 \
    --nat-gateway-id $(echo ${igw} | jq -r '.InternetGateway.InternetGatewayId') \
    > /dev/null 2>&1

$ for i in $(seq 1 3); do
    j="subnet${i}_public"
    aws ec2 associate-route-table \
        --route-table-id $(echo ${routetable0} | jq -r '.RouteTable.RouteTableId') \
        --subnet-id $(echo ${!j} | jq -r '.Subnet.SubnetId')
done

$ for i in $(seq 1 3); do
    export routetable${i}="$(aws ec2 create-route-table \
        --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId') \
    )"
done

$ for i in $(seq 1 3); do
    j="routetable${i}"
    k="natgw${i}"
    aws ec2 create-route \
        --route-table-id $(echo ${!j} | jq -r '.RouteTable.RouteTableId') \
        --destination-cidr-block 0.0.0.0/0 \
        --nat-gateway-id $(echo ${!k} | jq -r '.NatGateway.NatGatewayId') \
        > /dev/null 2>&1
done

$ for i in $(seq 1 3); do
    j="routetable${i}"
    k="subnet${i}_private"
    aws ec2 associate-route-table \
        --route-table-id $(echo ${!j} | jq -r '.RouteTable.RouteTableId') \
        --subnet-id $(echo ${!k} | jq -r '.Subnet.SubnetId')
done

Deploy SecurityGroups and rules

$ for i in Bastion infra master node; do
    export awssg_$(echo ${i} | tr A-Z a-z)="$(aws ec2 create-security-group \
    --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId') \
    --group-name ${i} \
    --description ${i})"
done

$ aws ec2 authorize-security-group-ingress \
    --group-id $(echo ${awssg_bastion} | jq -r '.GroupId') \
    --ip-permissions '[
                      {"IpProtocol": "icmp", "FromPort": 8, "ToPort": -1, "IpRanges": [{"CidrIp": "0.0.0.0/0"}]},
                      {"IpProtocol": "tcp", "FromPort": 22, "ToPort": 22, "IpRanges": [{"CidrIp": "0.0.0.0/0"}]}
                      ]'

$ aws ec2 authorize-security-group-ingress \
    --group-id $(echo ${awssg_infra} | jq -r '.GroupId') \
    --ip-permissions '[
                      {"IpProtocol": "tcp", "FromPort": 80, "ToPort": 80, "IpRanges": [{"CidrIp": "0.0.0.0/0"}]},
                      {"IpProtocol": "tcp", "FromPort": 443, "ToPort": 443, "IpRanges": [{"CidrIp": "0.0.0.0/0"}]}
                      ]'

$ aws ec2 authorize-security-group-ingress \
    --group-id $(echo ${awssg_master} | jq -r '.GroupId') \
    --ip-permissions '[
                      {"IpProtocol": "tcp", "FromPort": 443, "ToPort": 443, "IpRanges": [{"CidrIp": "0.0.0.0/0"}]}
                      ]'

$ for i in 2379-2380; do
    aws ec2 authorize-security-group-ingress \
    --group-id $(echo ${awssg_master} | jq -r '.GroupId') \
    --protocol tcp \
    --port $i \
    --source-group $(echo ${awssg_master} | jq -r '.GroupId')
done

$ for i in 2379-2380; do
    aws ec2 authorize-security-group-ingress \
    --group-id $(echo ${awssg_master} | jq -r '.GroupId') \
    --protocol tcp \
    --port $i \
    --source-group $(echo ${awssg_node} | jq -r '.GroupId')
done

$ aws ec2 authorize-security-group-ingress \
    --group-id $(echo ${awssg_node} | jq -r '.GroupId') \
    --ip-permissions '[
                      {"IpProtocol": "icmp", "FromPort": 8, "ToPort": -1, "IpRanges": [{"CidrIp": "0.0.0.0/0"}]}
                      ]'

$ aws ec2 authorize-security-group-ingress \
    --group-id $(echo ${awssg_node} | jq -r '.GroupId') \
    --protocol tcp \
    --port 22 \
    --source-group $(echo ${awssg_bastion} | jq -r '.GroupId')

$ for i in 53 2049 8053 10250; do
    aws ec2 authorize-security-group-ingress \
    --group-id $(echo ${awssg_node} | jq -r '.GroupId') \
    --protocol tcp \
    --port $i \
    --source-group $(echo ${awssg_node} | jq -r '.GroupId')
done

$ for i in 53 4789 8053; do
    aws ec2 authorize-security-group-ingress \
    --group-id $(echo ${awssg_node} | jq -r '.GroupId') \
    --protocol udp \
    --port $i \
    --source-group $(echo ${awssg_node} | jq -r '.GroupId')
done

Deploy ELB’s

$ export elb_masterext=$(aws elb create-load-balancer \
    --load-balancer-name ${clusterid}-public-master \
    --subnets \
      $(echo ${subnet1_public} | jq -r '.Subnet.SubnetId') \
      $(echo ${subnet2_public} | jq -r '.Subnet.SubnetId') \
      $(echo ${subnet3_public} | jq -r '.Subnet.SubnetId') \
    --listener Protocol=TCP,LoadBalancerPort=443,InstanceProtocol=TCP,InstancePort=443 \
    --security-groups $(echo ${awssg_master} | jq -r '.GroupId') \
    --scheme internet-facing \
    --tags Key=name,Value=${clusterid}-public-master Key=Clusterid,Value=${clusterid})

$ aws elb modify-load-balancer-attributes \
    --load-balancer-name ${clusterid}-public-master \
    --load-balancer-attributes "{
        \"CrossZoneLoadBalancing\":{\"Enabled\":true},
        \"ConnectionDraining\":{\"Enabled\":false}
    }"

$ aws elb configure-health-check \
    --load-balancer-name ${clusterid}-public-master \
    --health-check Target=HTTPS:443/api,HealthyThreshold=3,Interval=5,Timeout=2,UnhealthyThreshold=2

$ export elb_masterint=$(aws elb create-load-balancer \
    --load-balancer-name ${clusterid}-private-master \
    --subnets \
      $(echo ${subnet1_private} | jq -r '.Subnet.SubnetId') \
      $(echo ${subnet2_private} | jq -r '.Subnet.SubnetId') \
      $(echo ${subnet3_private} | jq -r '.Subnet.SubnetId') \
    --listener Protocol=TCP,LoadBalancerPort=443,InstanceProtocol=TCP,InstancePort=443 \
    --security-groups $(echo ${awssg_master} | jq -r '.GroupId') \
    --scheme internal \
    --tags Key=name,Value=${clusterid}-private-master Key=Clusterid,Value=${clusterid})

$ aws elb modify-load-balancer-attributes \
    --load-balancer-name ${clusterid}-private-master \
    --load-balancer-attributes "{
        \"CrossZoneLoadBalancing\":{\"Enabled\":true},
        \"ConnectionDraining\":{\"Enabled\":false}
    }"

$ aws elb configure-health-check \
    --load-balancer-name ${clusterid}-private-master \
    --health-check Target=HTTPS:443/api,HealthyThreshold=3,Interval=5,Timeout=2,UnhealthyThreshold=2

$ export elb_infraext=$(aws elb create-load-balancer \
    --load-balancer-name ${clusterid}-public-infra \
    --subnets \
      $(echo ${subnet1_public} | jq -r '.Subnet.SubnetId') \
      $(echo ${subnet2_public} | jq -r '.Subnet.SubnetId') \
      $(echo ${subnet3_public} | jq -r '.Subnet.SubnetId') \
    --listener \
      Protocol=TCP,LoadBalancerPort=80,InstanceProtocol=TCP,InstancePort=80 \
      Protocol=TCP,LoadBalancerPort=443,InstanceProtocol=TCP,InstancePort=443 \
    --security-groups $(echo ${awssg_infra} | jq -r '.GroupId') \
    --scheme internet-facing \
    --tags Key=name,Value=${clusterid}-public-infra Key=Clusterid,Value=${clusterid})

$ aws elb modify-load-balancer-attributes \
    --load-balancer-name ${clusterid}-public-infra \
    --load-balancer-attributes "{
        \"CrossZoneLoadBalancing\":{\"Enabled\":true},
        \"ConnectionDraining\":{\"Enabled\":false}
    }"

$ aws elb configure-health-check \
    --load-balancer-name ${clusterid}-public-infra \
    --health-check Target=TCP:443,HealthyThreshold=2,Interval=5,Timeout=2,UnhealthyThreshold=2

Deploy Route53 zones and resources

$ export route53_extzone=$(aws route53 create-hosted-zone \
  --caller-reference $(date +%s) \
  --name ${clusterid}.${dns_domain} \
  --hosted-zone-config "PrivateZone=False")

$ export route53_intzone=$(aws route53 create-hosted-zone \
  --caller-reference $(date +%s) \
  --name ${clusterid}.${dns_domain} \
  --vpc "VPCRegion=${region},VPCId=$(echo ${vpc} | jq -r '.Vpc.VpcId')" \
  --hosted-zone-config "PrivateZone=True")

$ aws route53 change-resource-record-sets \
    --hosted-zone-id $(echo ${route53_extzone} | jq -r '.HostedZone.Id' | sed 's/\/hostedzone\///g') \
    --change-batch "\
{ \
  \"Changes\": [ \
    { \
      \"Action\": \"CREATE\", \
      \"ResourceRecordSet\": { \
        \"Name\": \"master.${clusterid}.${dns_domain}\", \
        \"Type\": \"CNAME\", \
        \"TTL\": 300, \
        \"ResourceRecords\": [ \
          { \"Value\": \"$(echo ${elb_masterext} | jq -r '.DNSName')\" } \
        ] \
      } \
    } \
  ] \
}"

$ aws route53 change-resource-record-sets \
    --hosted-zone-id $(echo ${route53_intzone} | jq -r '.HostedZone.Id' | sed 's/\/hostedzone\///g') \
    --change-batch "\
{ \
  \"Changes\": [ \
    { \
      \"Action\": \"CREATE\", \
      \"ResourceRecordSet\": { \
        \"Name\": \"master.${clusterid}.${dns_domain}\", \
        \"Type\": \"CNAME\", \
        \"TTL\": 300, \
        \"ResourceRecords\": [ \
          { \"Value\": \"$(echo ${elb_masterint} | jq -r '.DNSName')\" } \
        ] \
      } \
    } \
  ] \
}"

$ aws route53 change-resource-record-sets \
    --hosted-zone-id $(echo ${route53_extzone} | jq -r '.HostedZone.Id' | sed 's/\/hostedzone\///g') \
    --change-batch "\
{ \
  \"Changes\": [ \
    { \
      \"Action\": \"CREATE\", \
      \"ResourceRecordSet\": { \
        \"Name\": \".apps.${clusterid}.${dns_domain}\", \ \"Type\": \"CNAME\", \ \"TTL\": 300, \ \"ResourceRecords\": [ \ { \"Value\": \"$(echo ${elb_infraext} | jq -r '.DNSName')\" } \ ] \ } \ } \ ] \ }" $ aws route53 change-resource-record-sets \ --hosted-zone-id $(echo ${route53_intzone} | jq -r '.HostedZone.Id' | sed 's/\/hostedzone\///g') \ --change-batch "\ { \ \"Changes\": [ \ { \ \"Action\": \"CREATE\", \ \"ResourceRecordSet\": { \ \"Name\": \".apps.${clusterid}.${dns_domain}\", \
        \"Type\": \"CNAME\", \
        \"TTL\": 300, \
        \"ResourceRecords\": [ \
          { \"Value\": \"$(echo ${elb_infraext} | jq -r '.DNSName')\" } \
        ] \
      } \
    } \
  ] \
}"

Create EC2 user-data script

$ cat << EOF > /tmp/ec2_userdata.sh
#!/bin/bash
if [ "\$#" -ne 2 ]; then exit 2; fi

printf '%s\n' "#cloud-config"

printf '%s' "cloud_config_modules:"
if [ "\$1" == 'bastion' ]; then
  printf '\n%s\n\n' "- package-update-upgrade-install"
else
  printf '\n%s\n\n' "- package-update-upgrade-install
- disk_setup
- mounts
- cc_write_files"
fi

printf '%s' "packages:"
if [ "\$1" == 'bastion' ]; then
  printf '\n%s\n' "- nmap-ncat"
else
  printf '\n%s\n' "- lvm2"
fi

if [ "\$1" != 'bastion' ]; then
  printf '\n%s' 'write_files:
- content: |
    STORAGE_DRIVER=overlay2
    DEVS=/dev/'
  if [[ "\$2" =~ (c5|c5d|i3.metal|m5) ]]; then
    printf '%s' 'nvme1n1'
  else
    printf '%s' 'xvdb'
  fi
  printf '\n%s\n' '    VG=dockervg
    CONTAINER_ROOT_LV_NAME=dockerlv
    CONTAINER_ROOT_LV_MOUNT_PATH=/var/lib/docker
    CONTAINER_ROOT_LV_SIZE=100%FREE
  path: "/etc/sysconfig/docker-storage-setup"
  permissions: "0644"
  owner: "root"'

  printf '\n%s' 'fs_setup:'
  printf '\n%s' '- label: ocp_emptydir
  filesystem: xfs
  device: /dev/'
  if [[ "\$2" =~ (c5|c5d|i3.metal|m5) ]]; then
    printf '%s\n' 'nvme2n1'
  else
    printf '%s\n' 'xvdc'
  fi
  printf '%s' '  partition: auto'
  if [ "\$1" == 'master' ]; then
    printf '\n%s' '- label: etcd
  filesystem: xfs
  device: /dev/'
    if [[ "\$2" =~ (c5|c5d|i3.metal|m5) ]]; then
      printf '%s\n' 'nvme3n1'
    else
      printf '%s\n' 'xvdd'
    fi
    printf '%s' '  partition: auto'
  fi
  printf '\n'

  printf '\n%s' 'mounts:'
  printf '\n%s' '- [ "LABEL=ocp_emptydir", "/var/lib/origin/openshift.local.volumes", xfs, "defaults,gquota" ]'
  if [ "\$1" == 'master' ]; then
    printf '\n%s' '- [ "LABEL=etcd", "/var/lib/etcd", xfs, "defaults,gquota" ]'
  fi
  printf '\n'
fi
EOF

$ chmod u+x /tmp/ec2_userdata.sh

Deploy the bastion EC2, sleep for 15 sec while AWS instantiates the EC2, and associate an EIP with the bastion for direct Internet access.

$ export ec2_bastion=$(aws ec2 run-instances \
    --image-id ${ec2ami[1]} \
    --count 1 \
    --instance-type ${ec2_type_bastion} \
    --key-name ${clusterid}.${dns_domain} \
    --security-group-ids $(echo ${awssg_bastion} | jq -r '.GroupId') \
    --subnet-id $(echo ${subnet1_public} | jq -r '.Subnet.SubnetId') \
    --associate-public-ip-address \
    --block-device-mappings "DeviceName=/dev/sda1,Ebs={DeleteOnTermination=False,VolumeSize=25}" \
    --user-data "$(/tmp/ec2_userdata.sh bastion ${ec2_type_bastion})" \
    --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=bastion},{Key=Clusterid,Value=${clusterid}}]" \
  )

$ sleep 15

$ aws ec2 associate-address \
    --allocation-id $(echo ${eip0} | jq -r '.AllocationId') \
    --instance-id $(echo ${ec2_bastion} | jq -r '.Instances[].InstanceId')

Create the master, infra and node EC2s along with EBS volumes

$ for i in $(seq 1 3); do
    j="subnet${i}_private"
    export ec2_master${i}="$(aws ec2 run-instances \
        --image-id ${ec2ami[1]} \
        --count 1 \
        --instance-type ${ec2_type_master} \
        --key-name ${clusterid}.${dns_domain} \
        --security-group-ids $(echo ${awssg_master} | jq -r '.GroupId') $(echo ${awssg_node} | jq -r '.GroupId') \
        --subnet-id $(echo ${!j} | jq -r '.Subnet.SubnetId') \
        --block-device-mappings \
            "DeviceName=/dev/sda1,Ebs={DeleteOnTermination=False,VolumeSize=100}" \
            "DeviceName=/dev/xvdb,Ebs={DeleteOnTermination=False,VolumeSize=100}" \
            "DeviceName=/dev/xvdc,Ebs={DeleteOnTermination=False,VolumeSize=100}" \
            "DeviceName=/dev/xvdd,Ebs={DeleteOnTermination=False,VolumeSize=100}" \
        --user-data "$(/tmp/ec2_userdata.sh master ${ec2_type_master})" \
        --tag-specifications "ResourceType=instance,Tags=[ \
            {Key=Name,Value=master${i}}, \
            {Key=Clusterid,Value=${clusterid}}, \
            {Key=ami,Value=${ec2ami}}, \
            {Key=kubernetes.io/cluster/${clusterid},Value=${clusterid}}]"
        )"
done

$ for i in $(seq 1 3); do
    j="subnet${i}_private"
    export ec2_infra${i}="$(aws ec2 run-instances \
        --image-id ${ec2ami[1]} \
        --count 1 \
        --instance-type ${ec2_type_infra} \
        --key-name ${clusterid}.${dns_domain} \
        --security-group-ids $(echo ${awssg_infra} | jq -r '.GroupId') $(echo ${awssg_node} | jq -r '.GroupId') \
        --subnet-id $(echo ${!j} | jq -r '.Subnet.SubnetId') \
        --block-device-mappings \
            "DeviceName=/dev/sda1,Ebs={DeleteOnTermination=False,VolumeSize=100}" \
            "DeviceName=/dev/xvdb,Ebs={DeleteOnTermination=False,VolumeSize=100}" \
            "DeviceName=/dev/xvdc,Ebs={DeleteOnTermination=False,VolumeSize=100}" \
            "DeviceName=/dev/xvdd,Ebs={DeleteOnTermination=False,VolumeSize=100}" \
        --user-data "$(/tmp/ec2_userdata.sh infra ${ec2_type_infra})" \
        --tag-specifications "ResourceType=instance,Tags=[ \
            {Key=Name,Value=infra${i}}, \
            {Key=Clusterid,Value=${clusterid}}, \
            {Key=ami,Value=${ec2ami}}, \
            {Key=kubernetes.io/cluster/${clusterid},Value=${clusterid}}]"
        )"
done

$ for i in $(seq 1 3); do
    j="subnet${i}_private"
    export ec2_node${i}="$(aws ec2 run-instances \
        --image-id ${ec2ami[1]} \
        --count 1 \
        --instance-type ${ec2_type_node} \
        --key-name ${clusterid}.${dns_domain} \
        --security-group-ids $(echo ${awssg_node} | jq -r '.GroupId') \
        --subnet-id $(echo ${!j} | jq -r '.Subnet.SubnetId') \
        --block-device-mappings \
            "DeviceName=/dev/sda1,Ebs={DeleteOnTermination=False,VolumeSize=100}" \
            "DeviceName=/dev/xvdb,Ebs={DeleteOnTermination=False,VolumeSize=100}" \
            "DeviceName=/dev/xvdc,Ebs={DeleteOnTermination=False,VolumeSize=100}" \
        --user-data "$(/tmp/ec2_userdata.sh node ${ec2_type_node})" \
        --tag-specifications "ResourceType=instance,Tags=[ \
            {Key=Name,Value=node${i}}, \
            {Key=Clusterid,Value=${clusterid}}, \
            {Key=ami,Value=${ec2ami}}, \
            {Key=kubernetes.io/cluster/${clusterid},Value=${clusterid}}]"
        )"
done

$ rm -rf /tmp/ec2_userdata*

Register EC2s to ELB’s

$ export elb_masterextreg=$(aws elb register-instances-with-load-balancer \
    --load-balancer-name ${clusterid}-public-master \
    --instances \
      $(echo ${ec2_master1} | jq -r '.Instances[].InstanceId') \
      $(echo ${ec2_master2} | jq -r '.Instances[].InstanceId') \
      $(echo ${ec2_master3} | jq -r '.Instances[].InstanceId') \
  )

$ export elb_masterintreg=$(aws elb register-instances-with-load-balancer \
    --load-balancer-name ${clusterid}-private-master \
    --instances \
      $(echo ${ec2_master1} | jq -r '.Instances[].InstanceId') \
      $(echo ${ec2_master2} | jq -r '.Instances[].InstanceId') \
      $(echo ${ec2_master3} | jq -r '.Instances[].InstanceId') \
  )

$ export elb_infrareg=$(aws elb register-instances-with-load-balancer \
    --load-balancer-name ${clusterid}-public-infra \
    --instances \
      $(echo ${ec2_infra1} | jq -r '.Instances[].InstanceId') \
      $(echo ${ec2_infra2} | jq -r '.Instances[].InstanceId') \
      $(echo ${ec2_infra3} | jq -r '.Instances[].InstanceId') \
  )

Create tags on AWS components

 aws ec2 create-tags --resources $(echo $vpc | jq -r ".Vpc.VpcId") --tags Key=Name,Value=${clusterid}; \
 aws ec2 create-tags --resources $(echo ${eip0} | jq -r '.AllocationId') --tags Key=Name,Value=bastion; \
 aws ec2 create-tags --resources $(echo ${eip1} | jq -r '.AllocationId') --tags Key=Name,Value=${az[0]}; \
 aws ec2 create-tags --resources $(echo ${eip2} | jq -r '.AllocationId') --tags Key=Name,Value=${az[1]}; \
 aws ec2 create-tags --resources $(echo ${eip3} | jq -r '.AllocationId') --tags Key=Name,Value=${az[2]}; \
 aws ec2 create-tags --resources $(echo ${natgw1} | jq -r '.NatGateway.NatGatewayId') --tags Key=Name,Value=${az[0]}; \
 aws ec2 create-tags --resources $(echo ${natgw2} | jq -r '.NatGateway.NatGatewayId') --tags Key=Name,Value=${az[1]}; \
 aws ec2 create-tags --resources $(echo ${natgw3} | jq -r '.NatGateway.NatGatewayId') --tags Key=Name,Value=${az[2]}; \
 aws ec2 create-tags --resources $(echo ${routetable0} | jq -r '.RouteTable.RouteTableId') --tags Key=Name,Value=routing; \
 aws ec2 create-tags --resources $(echo ${routetable1} | jq -r '.RouteTable.RouteTableId') --tags Key=Name,Value=${az[0]}; \
 aws ec2 create-tags --resources $(echo ${routetable2} | jq -r '.RouteTable.RouteTableId') --tags Key=Name,Value=${az[1]}; \
 aws ec2 create-tags --resources $(echo ${routetable3} | jq -r '.RouteTable.RouteTableId') --tags Key=Name,Value=${az[2]}; \
 aws ec2 create-tags --resources $(echo ${awssg_bastion} | jq -r '.GroupId') --tags Key=Name,Value=Bastion; \
 aws ec2 create-tags --resources $(echo ${awssg_bastion} | jq -r '.GroupId') --tags Key=clusterid,Value=${clusterid}; \
 aws ec2 create-tags --resources $(echo ${awssg_master} | jq -r '.GroupId') --tags Key=Name,Value=Master; \
 aws ec2 create-tags --resources $(echo ${awssg_master} | jq -r '.GroupId') --tags Key=clusterid,Value=${clusterid}; \
 aws ec2 create-tags --resources $(echo ${awssg_infra} | jq -r '.GroupId') --tags Key=Name,Value=Infra; \
 aws ec2 create-tags --resources $(echo ${awssg_infra} | jq -r '.GroupId') --tags Key=clusterid,Value=${clusterid}; \
 aws ec2 create-tags --resources $(echo ${awssg_node} | jq -r '.GroupId') --tags Key=Name,Value=Node; \
 aws ec2 create-tags --resources $(echo ${awssg_node} | jq -r '.GroupId') --tags Key=clusterid,Value=${clusterid}

Create configuration files. These files are used to assist with installing

$ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain}
#<!-- BEGIN OUTPUT -->
Host bastion
  HostName                 $(echo ${eip0} | jq -r '.PublicIp')
  User                     ec2-user
  StrictHostKeyChecking    no
  ProxyCommand             none
  CheckHostIP              no
  ForwardAgent             yes
  ServerAliveInterval      15
  TCPKeepAlive             yes
  ControlMaster            auto
  ControlPath              ~/.ssh/mux-%r@%h:%p
  ControlPersist           15m
  ServerAliveInterval      30
  IdentityFile             ~/.ssh/${clusterid}.${dns_domain}

Host *.compute-1.amazonaws.com
  ProxyCommand             ssh -w 300 -W %h:%p bastion
  user                     ec2-user
  StrictHostKeyChecking    no
  CheckHostIP              no
  ServerAliveInterval      30
  IdentityFile             ~/.ssh/${clusterid}.${dns_domain}

Host *.ec2.internal
  ProxyCommand             ssh -w 300 -W %h:%p bastion
  user                     ec2-user
  StrictHostKeyChecking    no
  CheckHostIP              no
  ServerAliveInterval      30
  IdentityFile             ~/.ssh/${clusterid}.${dns_domain}
#<!-- END OUTPUT -->
EOF

$ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain}-domaindelegation
#<!-- BEGIN OUTPUT -->
$(echo ${route53_extzone} | jq -r '.DelegationSet.NameServers[]')
#<!-- END OUTPUT -->
EOF

$ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain}-cpkuser_access_key
#<!-- BEGIN OUTPUT -->
openshift_cloudprovider_aws_access_key=$(echo ${iamuser_accesskey} | jq -r '.AccessKey.AccessKeyId')
openshift_cloudprovider_aws_secret_key=$(echo ${iamuser_accesskey} | jq -r '.AccessKey.SecretAccessKey')
#<!-- END OUTPUT -->
EOF

$ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain}-cpk
#<!-- BEGIN OUTPUT -->
openshift_cloudprovider_kind=aws
openshift_clusterid=${clusterid}
EOF
cat ~/.ssh/config-${clusterid}.${dns_domain}-cpkuser_access_key | \
    grep -v 'OUTPUT -->' >> \
    ~/.ssh/config-${clusterid}.${dns_domain}-cpk
cat << EOF >> ~/.ssh/config-${clusterid}.${dns_domain}-cpk
#<!-- END OUTPUT -->
EOF

$ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain}-s3user_access_key
#<!-- BEGIN OUTPUT -->
openshift_hosted_registry_storage_s3_accesskey=$(echo ${s3user_accesskey} | jq -r '.AccessKey.AccessKeyId')
openshift_hosted_registry_storage_s3_secretkey=$(echo ${s3user_accesskey} | jq -r '.AccessKey.SecretAccessKey')
#<!-- END OUTPUT -->
EOF

$ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain}-s3
#<!-- BEGIN OUTPUT -->
openshift_hosted_manage_registry=true
openshift_hosted_registry_storage_kind=object
openshift_hosted_registry_storage_provider=s3
EOF
cat ~/.ssh/config-${clusterid}.${dns_domain}-s3user_access_key | \
    grep -v 'OUTPUT -->' >> \
    ~/.ssh/config-${clusterid}.${dns_domain}-s3
cat << EOF >> ~/.ssh/config-${clusterid}.${dns_domain}-s3
openshift_hosted_registry_storage_s3_bucket=${clusterid}.${dns_domain}-registry
openshift_hosted_registry_storage_s3_region=${region}
openshift_hosted_registry_storage_s3_chunksize=26214400
openshift_hosted_registry_storage_s3_rootdirectory=/registry
openshift_hosted_registry_pullthrough=true
openshift_hosted_registry_acceptschema2=true
openshift_hosted_registry_enforcequota=true
openshift_hosted_registry_replicas=3
openshift_hosted_registry_selector='region=infra'
#<!-- END OUTPUT -->
EOF

$ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain}-urls
#<!-- BEGIN OUTPUT -->
openshift_master_default_subdomain=apps.${clusterid}.${dns_domain}
openshift_master_cluster_hostname=master.${clusterid}.${dns_domain}
openshift_master_cluster_public_hostname=master.${clusterid}.${dns_domain}
#<!-- END OUTPUT -->
EOF

$ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain}-hosts
[masters]
$(echo ${ec2_master1} | jq -r '.Instances[].PrivateDnsName') openshift_node_labels="{'region': 'master'}"
$(echo ${ec2_master2} | jq -r '.Instances[].PrivateDnsName') openshift_node_labels="{'region': 'master'}"
$(echo ${ec2_master3} | jq -r '.Instances[].PrivateDnsName') openshift_node_labels="{'region': 'master'}"

[etcd]

[etcd:children]
masters

[nodes]
$(echo ${ec2_node1} | jq -r '.Instances[].PrivateDnsName') openshift_node_labels="{'region': 'apps'}"
$(echo ${ec2_node2} | jq -r '.Instances[].PrivateDnsName') openshift_node_labels="{'region': 'apps'}"
$(echo ${ec2_node3} | jq -r '.Instances[].PrivateDnsName') openshift_node_labels="{'region': 'apps'}"
$(echo ${ec2_infra1} | jq -r '.Instances[].PrivateDnsName') openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
$(echo ${ec2_infra2} | jq -r '.Instances[].PrivateDnsName') openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
$(echo ${ec2_infra3} | jq -r '.Instances[].PrivateDnsName') openshift_node_labels="{'region': 'infra', 'zone': 'default'}"

[nodes:children]
masters
EOF

2.11.2. Ansible

GitHub OpenShift openshift-ansible-contrib repository contains an Ansible playbook that provides a friendly and repeatable deployment experience.

Clone openshift-ansible-contrib repository then enter the reference architecture 3.9 directory.

$ git clone https://github.com/openshift/openshift-ansible-contrib.git

$ cd openshift-ansible-contrib/reference-architecture/3.9

Create a simple Ansible inventory

$ sudo vi /etc/ansible/hosts
[local]
127.0.0.1

[local:vars]
ansible_connection=local
ansible_become=False

Review playbooks/vars/main.yaml now and ensure values fit requirements.

When values are satisfactory execute deploy_aws.yaml play to deploy AWS infrastructure.

$ ansible-playbook playbooks/deploy_aws.yaml
Note

Troubleshooting the deploy_aws.yaml play is simple. It is self repairing. If any errors are encountered simply rerun play.

2.12. Create AWS Infrastructure for Red Hat OpenShift Container Platform CNS (Optional)

2.12.1. CLI

Set environment variables to be sourced by other commands. These values can be modified to fit any environment.

$ export ec2_type_cns="m5.2xlarge"

Deploy SecurityGroup and rules

$ export awssg_cns=$(aws ec2 create-security-group \
    --vpc-id $(echo ${vpc} | jq -r '.Vpc.VpcId') \
    --group-name cns \
    --description "cns")

$ for i in 111 2222 3260 24007-24008 24010 49152-49664; do
    aws ec2 authorize-security-group-ingress \
    --group-id $(echo ${awssg_cns} | jq -r '.GroupId') \
    --protocol tcp \
    --port $i \
    --source-group $(echo ${awssg_cns} | jq -r '.GroupId')
done

$ for i in 3260 24007-24008 24010 49152-49664; do
    aws ec2 authorize-security-group-ingress \
    --group-id $(echo ${awssg_cns} | jq -r '.GroupId') \
    --protocol tcp \
    --port $i \
    --source-group $(echo ${awssg_node} | jq -r '.GroupId')
done

$ aws ec2 authorize-security-group-ingress \
    --group-id $(echo ${awssg_cns} | jq -r '.GroupId') \
    --protocol udp \
    --port 111 \
    --source-group $(echo ${awssg_cns} | jq -r '.GroupId')

Create node EC2 instances along with EBS volumes

$ for i in 1 2 3; do
    j="subnet${i}_private"
    export ec2_cns${i}="$(aws ec2 run-instances \
        --image-id ${ec2ami[1]} \
        --count 1 \
        --instance-type ${ec2_type_cns} \
        --key-name ${clusterid}.${dns_domain} \
        --security-group-ids $(echo ${awssg_cns} | jq -r '.GroupId') $(echo ${awssg_node} | jq -r '.GroupId') \
        --subnet-id $(echo ${!j} | jq -r '.Subnet.SubnetId') \
        --block-device-mappings \
            "DeviceName=/dev/sda1,Ebs={DeleteOnTermination=False,VolumeSize=100}" \
            "DeviceName=/dev/xvdb,Ebs={DeleteOnTermination=False,VolumeSize=100}" \
            "DeviceName=/dev/xvdc,Ebs={DeleteOnTermination=False,VolumeSize=100}" \
            "DeviceName=/dev/xvdd,Ebs={DeleteOnTermination=False,VolumeSize=100}" \
        --user-data "$(/tmp/ec2_userdata.sh node ${ec2_type_node})" \
        --tag-specifications "ResourceType=instance,Tags=[ \
            {Key=Name,Value=cns${i}}, \
            {Key=Clusterid,Value=${clusterid}}, \
            {Key=ami,Value=${ec2ami}}, \
            {Key=kubernetes.io/cluster/${clusterid},Value=${clusterid}}]"
        )"
done

Create configuration files. These files are used to assist with installing Red Hat OpenShift Container Platform.

$ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain}-hostscns
$(echo ${ec2_cns1} | jq -r '.Instances[].PrivateDnsName') openshift_schedulable=True
$(echo ${ec2_cns2} | jq -r '.Instances[].PrivateDnsName') openshift_schedulable=True
$(echo ${ec2_cns3} | jq -r '.Instances[].PrivateDnsName') openshift_schedulable=True

EOF

$ cat << EOF > ~/.ssh/config-${clusterid}.${dns_domain}-hostsgfs
[glusterfs]
$(echo ${ec2_cns1} | jq -r '.Instances[].PrivateDnsName') glusterfs_devices='[ "/dev/nvme3n1" ]'
$(echo ${ec2_cns2} | jq -r '.Instances[].PrivateDnsName') glusterfs_devices='[ "/dev/nvme3n1" ]'
$(echo ${ec2_cns3} | jq -r '.Instances[].PrivateDnsName') glusterfs_devices='[ "/dev/nvme3n1" ]'
EOF

2.12.2. Ansible

Run deploy_aws_cns.yaml play to deploy additional infrastructure for CNS.

$ ansible-playbook playbooks/deploy_aws_cns.yaml
Note

deploy_aws_cns.yaml playbook requires deploy_aws.yaml to be previously run. Troubleshooting the deploy_aws_cns.yaml play is simple. It is self repairing. If any errors are encountered simply rerun play.

2.13. Bastion Configuration

It is often preferrable to use a bastion host as an ssh proxy for security. Awscli and Ansible both output a ssh configuration file to enable direct ssh connections to Red Hat OpenShift Container Platform instances.

Perform the following procedure to set configuration:

$ mv ~/.ssh/config ~/.ssh/config-orig

$ ln -s ~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN > ~/.ssh/config

$ chmod 400 ~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN >

Perform the following procedure to ensure ssh-agent is running and local ssh key can be used to proxy connections to target EC2 instances:

$ if [ ! "$(env | grep SSH_AGENT_PID)" ] || [ ! "$(ps -ef | grep -v grep | grep ${SSH_AGENT_PID})" ]; then
  rm -rf ${SSH_AUTH_SOCK} 2> /dev/null
  unset SSH_AUTH_SOCK
  unset SSH_AGENT_PID
  pkill ssh-agent
  export sshagent=$(nohup ssh-agent &)
  export sshauthsock=$(echo ${sshagent} | awk -F'; ' {'print $1'})
  export sshagentpid=$(echo ${sshagent} | awk -F'; ' {'print $3'})
  export ${sshauthsock}
  export ${sshagentpid}
  for i in sshagent sshauthsock sshagentpid; do
    unset $i
  done
fi

$ export sshkey=($(cat ~/.ssh/< CLUSTERID >.< DNS_DOMAIN >.pub))

$ IFS=$'\n'; if [ ! $(ssh-add -L | grep ${sshkey[1]}) ]; then
  ssh-add ~/.ssh/< CLUSTERID >.< DNS_DOMAIN >
fi

$ unset IFS

Verify < CLUSTERID >.< DNS_DOMAIN > ssh key is added to ssh-agent

$ ssh-add -l

$ ssh-add -L

2.14. OpenShift Preparations

Once the instances have been deployed and the ~/.ssh/config file reflects the deployment the following steps should be performed to prepare for the installation of OpenShift.

2.14.1. Public domain delegation

To access Red Hat OpenShift Container Platform on AWS from the public Internet delegation for the subdomain must be setup using the following information.

DNS subdomainRoute53 public zone NS records

< CLUSTERID >.< DNS_DOMAIN >

See file ~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN >-domaindelegation for correct values

WARN: DNS service provder domain delegation is out of scope of this guide. A customer will need to follow their providers best practise in delegation setup.

2.14.2. OpenShift Authentication

Red Hat OpenShift Container Platform provides the ability to use many different authentication platforms.

Detailed listing of other authentication providers are available at Configuring Authentication and User Agent.

For this reference architecture Google’s OpenID Connect Integration is used. When configuring the authentication, the following parameters must be added to the ansible inventory. An example is shown below.

openshift_master_identity_providers=[{'name': 'google', 'challenge': 'false', 'login': 'true', 'kind': 'GoogleIdentityProvider', 'mapping_method': 'claim', 'clientID': '246358064255-5ic2e4b1b9ipfa7hddfkhuf8s6eq2rfj.apps.googleusercontent.com', 'clientSecret': 'Za3PWZg7gQxM26HBljgBMBBF', 'hostedDomain': 'redhat.com'}]

2.14.3. Openshift-ansible Installer Inventory

This section provides an example inventory file required for an advanced installation of Red Hat OpenShift Container Platform.

The inventory file contains both variables and instances used for the configuration and deployment of Red Hat OpenShift Container Platform.

$ sudo vi /etc/ansible/hosts
[OSEv3:children]
masters
etcd
nodes

[OSEv3:vars]
debug_level=2
ansible_user=ec2-user
ansible_become=yes
openshift_deployment_type=openshift-enterprise
openshift_release=v3.9
openshift_master_api_port=443
openshift_master_console_port=443
openshift_portal_net=172.30.0.0/16
os_sdn_network_plugin_name='redhat/openshift-ovs-networkpolicy'
openshift_master_cluster_method=native
container_runtime_docker_storage_setup_device=/dev/nvme1n1
openshift_node_local_quota_per_fsgroup=512Mi
osm_use_cockpit=true
openshift_hostname_check=false
openshift_examples_modify_imagestreams=true
oreg_url=registry.access.redhat.com/openshift3/ose-${component}:${version}

openshift_hosted_router_selector='region=infra'
openshift_hosted_router_replicas=3

# UPDATE TO CORRECT IDENTITY PROVIDER
openshift_master_identity_providers=[{'name': 'google', 'challenge': 'false', 'login': 'true', 'kind': 'GoogleIdentityProvider', 'mapping_method': 'claim', 'clientID': '246358064255-5ic2e4b1b9ipfa7hddfkhuf8s6eq2rfj.apps.googleusercontent.com', 'clientSecret': 'Za3PWZg7gQxM26HBljgBMBBF', 'hostedDomain': 'redhat.com'}]

# UPDATE USING VALUES FOUND IN awscli env vars or ~/playbooks/var/main.yaml
# SEE ALSO FILE ~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN >-urls
openshift_master_default_subdomain=apps.< CLUSTERID >.< DNS_DOMAIN >
openshift_master_cluster_hostname=master.< CLUSTERID >.< DNS_DOMAIN >
openshift_master_cluster_public_hostname=master.< CLUSTERID >.< DNS_DOMAIN >

# UPDATE USING VALUES FOUND IN FILE ~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN >-cpk
openshift_cloudprovider_kind=aws
openshift_clusterid=refarch
openshift_cloudprovider_aws_access_key=UPDATEACCESSKEY
openshift_cloudprovider_aws_secret_key=UPDATESECRETKEY

# UPDATE USING VALUES FOUND IN FILE ~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN >-s3
openshift_hosted_manage_registry=true
openshift_hosted_registry_storage_kind=object
openshift_hosted_registry_storage_provider=s3
openshift_hosted_registry_storage_s3_accesskey=UPDATEACCESSKEY
openshift_hosted_registry_storage_s3_secretkey=UPDATESECRETKEY
openshift_hosted_registry_storage_s3_bucket=refarch-registry
openshift_hosted_registry_storage_s3_region=REGION
openshift_hosted_registry_storage_s3_chunksize=26214400
openshift_hosted_registry_storage_s3_rootdirectory=/registry
openshift_hosted_registry_pullthrough=true
openshift_hosted_registry_acceptschema2=true
openshift_hosted_registry_enforcequota=true
openshift_hosted_registry_replicas=3
openshift_hosted_registry_selector='region=infra'

# Aggregated logging
openshift_logging_install_logging=true
openshift_logging_storage_kind=dynamic
openshift_logging_storage_volume_size=25Gi
openshift_logging_es_cluster_size=3

# Metrics
openshift_metrics_install_metrics=true
openshift_metrics_storage_kind=dynamic
openshift_metrics_storage_volume_size=25Gi

openshift_enable_service_catalog=true

template_service_broker_install=true

# UPDATE USING HOSTS FOUND IN FILE ~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN >-hosts
[masters]
ip-172-16-30-15.ec2.internal openshift_node_labels="{'region': 'master'}"
ip-172-16-47-176.ec2.internal openshift_node_labels="{'region': 'master'}"
ip-172-16-48-251.ec2.internal openshift_node_labels="{'region': 'master'}"

[etcd]

[etcd:children]
masters

# UPDATE USING HOSTS FOUND IN FILE ~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN >-hosts
[nodes]
ip-172-16-16-239.ec2.internal openshift_node_labels="{'region': 'apps'}"
ip-172-16-37-179.ec2.internal openshift_node_labels="{'region': 'apps'}"
ip-172-16-60-134.ec2.internal openshift_node_labels="{'region': 'apps'}"
ip-172-16-17-209.ec2.internal openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
ip-172-16-46-136.ec2.internal openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
ip-172-16-56-149.ec2.internal openshift_node_labels="{'region': 'infra', 'zone': 'default'}"

[nodes:children]
masters
Note

Advanced installations example configurations and options are provided in /usr/share/doc/openshift-ansible-docs-3.9.*/docs/example-inventories directory.

2.14.4. CNS Inventory (Optional)

If CNS is used in the OpenShift installation specific variables must be set in the inventory.

$ sudo vi /etc/ansible/hosts
# ADD glusterfs to section [OSEv3:children]
[OSEv3:children]
masters
etcd
nodes
glusterfs

[OSEv3:vars]

....omitted...

# CNS storage cluster
openshift_storage_glusterfs_namespace=glusterfs
openshift_storage_glusterfs_storageclass=true
openshift_storage_glusterfs_storageclass_default=false
openshift_storage_glusterfs_block_deploy=false
openshift_storage_glusterfs_block_host_vol_create=false
openshift_storage_glusterfs_block_host_vol_size=80
openshift_storage_glusterfs_block_storageclass=false
openshift_storage_glusterfs_block_storageclass_default=false

....omitted...

# ADD CNS HOSTS TO SECTION [nodes]
# SEE INVENTORY IN FILE ~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN >-hostscns
[nodes]
....omitted...
ip-172-16-17-130.ec2.internal openshift_schedulable=True
ip-172-16-40-219.ec2.internal openshift_schedulable=True
ip-172-16-63-212.ec2.internal openshift_schedulable=True

# ADD SECTION [glusterfs]
# SEE INVENTORY IN FILE ~/.ssh/config-< CLUSTERID >.< DNS_DOMAIN >-hostsgfs
[glusterfs]
ip-172-16-17-130.ec2.internal glusterfs_devices='[ "/dev/nvme3n1" ]'
ip-172-16-40-219.ec2.internal glusterfs_devices='[ "/dev/nvme3n1" ]'
ip-172-16-63-212.ec2.internal glusterfs_devices='[ "/dev/nvme3n1" ]'

2.14.5. EC2 instances

2.14.5.1. Test network connectivity

Ansible ping is used to test if the local environment can reach all instances. If there is a failure here then please check ssh config or network connectivity for the instance.

$ ansible nodes -b -m ping

2.14.5.2. Remove RHUI yum repos

Existing Red Hat Update Infrastructure yum repos must be disabled to avoid repository and rpm dependency conflicts with Red Hat OpenShift Container Platform repository and rpms.

$ ansible nodes -b -m command -a "yum-config-manager \
    --disable 'rhui-REGION-client-config-server-7' \
    --disable 'rhui-REGION-rhel-server-rh-common' \
    --disable 'rhui-REGION-rhel-server-releases'"

2.14.5.3. Node Registration

Now that the inventory has been created the nodes must be subscribed using subscription-manager.

The ad-hoc playbook below uses the redhat_subscription module to register the instances. The first example uses the numeric pool value for the OpenShift subscription. The second uses an activation key and organization.

$ ansible nodes -b -m redhat_subscription -a \
    "state=present username=USER password=PASSWORD pool_ids=NUMBERIC_POOLID"

OR

$ ansible nodes -b -m redhat_subscription -a \
    "state=present activationkey=KEY org_id=ORGANIZATION"

2.14.5.4. Repository Setup

Once the instances are registered the proper repositories must be assigned to the instances to allow for packages for Red Hat OpenShift Container Platform to be installed.

$ ansible nodes -b -m shell -a \
    'subscription-manager repos --disable="*" \
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-extras-rpms" \
    --enable="rhel-7-server-ose-3.9-rpms" \
    --enable="rhel-7-fast-datapath-rpms"'

2.14.6. EmptyDir Storage

During the deployment of instances an extra volume is added to the instances for EmptyDir storage. EmptyDir provides ephemeral (as opposed to persistent) storage for containers. The volume is used to help ensure that the /var volume does not get filled by containers using the storage.

The user-data script attached to EC2 instances automatically provisioned filesystem, added an entry to fstab, and mounted this volume.

2.14.7. etcd Storage

During the deployment of the master instances an extra volume was added to the instances for etcd storage. Having the separate disks specifically for etcd ensures that all of the resources are available to the etcd service such as i/o and total disk space.

The user-data script attached to EC2 instances automatically provisioned filesystem, added an entry to fstab, and mounted this volume.

2.14.8. Container Storage

The prerequisite playbook provided by the OpenShift Ansible RPMs configures container storage and installs any remaining packages for the installation.

$ ansible-playbook \
    /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml