Menu Close

2.9.3.3. Other infrastructure components

  • A VPC
  • DNS entries
  • Load balancers (classic or network) and listeners
  • A public and a private Route 53 zone
  • Security groups
  • IAM roles
  • S3 buckets

If you are working in a disconnected environment, you are unable to reach the public IP addresses for EC2 and ELB endpoints. To resolve this, you must create a VPC endpoint and attach it to the subnet that the clusters are using. The endpoints should be named as follows:

  • ec2.<region>.amazonaws.com
  • elasticloadbalancing.<region>.amazonaws.com
  • s3.<region>.amazonaws.com

Required VPC components

You must provide a suitable VPC and subnets that allow communication to your machines.

ComponentAWS typeDescription

VPC

  • AWS::EC2::VPC
  • AWS::EC2::VPCEndpoint

You must provide a public VPC for the cluster to use. The VPC uses an endpoint that references the route tables for each subnet to improve communication with the registry that is hosted in S3.

Public subnets

  • AWS::EC2::Subnet
  • AWS::EC2::SubnetNetworkAclAssociation

Your VPC must have public subnets for between 1 and 3 availability zones and associate them with appropriate Ingress rules.

Internet gateway

  • AWS::EC2::InternetGateway
  • AWS::EC2::VPCGatewayAttachment
  • AWS::EC2::RouteTable
  • AWS::EC2::Route
  • AWS::EC2::SubnetRouteTableAssociation
  • AWS::EC2::NatGateway
  • AWS::EC2::EIP

You must have a public Internet gateway, with public routes, attached to the VPC. In the provided templates, each public subnet has a NAT gateway with an EIP address. These NAT gateways allow cluster resources, like private subnet instances, to reach the Internet and are not required for some restricted network or proxy scenarios.

Network access control

  • AWS::EC2::NetworkAcl
  • AWS::EC2::NetworkAclEntry

You must allow the VPC to access the following ports:

Port

Reason

80

Inbound HTTP traffic

443

Inbound HTTPS traffic

22

Inbound SSH traffic

1024 - 65535

Inbound ephemeral traffic

0 - 65535

Outbound ephemeral traffic

Private subnets

  • AWS::EC2::Subnet
  • AWS::EC2::RouteTable
  • AWS::EC2::SubnetRouteTableAssociation

Your VPC can have private subnets. The provided CloudFormation templates can create private subnets for between 1 and 3 availability zones. If you use private subnets, you must provide appropriate routes and tables for them.

Required DNS and load balancing components

Your DNS and load balancer configuration needs to use a public hosted zone and can use a private hosted zone similar to the one that the installation program uses if it provisions the cluster’s infrastructure. You must create a DNS entry that resolves to your load balancer. An entry for api.<cluster_name>.<domain> must point to the external load balancer, and an entry for api-int.<cluster_name>.<domain> must point to the internal load balancer.

The cluster also requires load balancers and listeners for port 6443, which are required for the Kubernetes API and its extensions, and port 22623, which are required for the Ignition config files for new machines. The targets will be the master nodes. Port 6443 must be accessible to both clients external to the cluster and nodes within the cluster. Port 22623 must be accessible to nodes within the cluster.

ComponentAWS typeDescription

DNS

AWS::Route53::HostedZone

The hosted zone for your internal DNS.

etcd record sets

AWS::Route53::RecordSet

The registration records for etcd for your control plane machines.

Public load balancer

AWS::ElasticLoadBalancingV2::LoadBalancer

The load balancer for your public subnets.

External API server record

AWS::Route53::RecordSetGroup

Alias records for the external API server.

External listener

AWS::ElasticLoadBalancingV2::Listener

A listener on port 6443 for the external load balancer.

External target group

AWS::ElasticLoadBalancingV2::TargetGroup

The target group for the external load balancer.

Private load balancer

AWS::ElasticLoadBalancingV2::LoadBalancer

The load balancer for your private subnets.

Internal API server record

AWS::Route53::RecordSetGroup

Alias records for the internal API server.

Internal listener

AWS::ElasticLoadBalancingV2::Listener

A listener on port 22623 for the internal load balancer.

Internal target group

AWS::ElasticLoadBalancingV2::TargetGroup

The target group for the internal load balancer.

Internal listener

AWS::ElasticLoadBalancingV2::Listener

A listener on port 6443 for the internal load balancer.

Internal target group

AWS::ElasticLoadBalancingV2::TargetGroup

The target group for the internal load balancer.

Security groups

The control plane and worker machines require access to the following ports:

GroupTypeIP ProtocolPort range

MasterSecurityGroup

AWS::EC2::SecurityGroup

icmp

0

tcp

22

tcp

6443

tcp

22623

WorkerSecurityGroup

AWS::EC2::SecurityGroup

icmp

0

tcp

22

BootstrapSecurityGroup

AWS::EC2::SecurityGroup

tcp

22

tcp

19531

Control plane Ingress

The control plane machines require the following Ingress groups. Each Ingress group is a AWS::EC2::SecurityGroupIngress resource.

Ingress groupDescriptionIP protocolPort range

MasterIngressEtcd

etcd

tcp

2379- 2380

MasterIngressVxlan

Vxlan packets

udp

4789

MasterIngressWorkerVxlan

Vxlan packets

udp

4789

MasterIngressInternal

Internal cluster communication and Kubernetes proxy metrics

tcp

9000 - 9999

MasterIngressWorkerInternal

Internal cluster communication

tcp

9000 - 9999

MasterIngressKube

Kubernetes kubelet, scheduler and controller manager

tcp

10250 - 10259

MasterIngressWorkerKube

Kubernetes kubelet, scheduler and controller manager

tcp

10250 - 10259

MasterIngressIngressServices

Kubernetes Ingress services

tcp

30000 - 32767

MasterIngressWorkerIngressServices

Kubernetes Ingress services

tcp

30000 - 32767

Worker Ingress

The worker machines require the following Ingress groups. Each Ingress group is a AWS::EC2::SecurityGroupIngress resource.

Ingress groupDescriptionIP protocolPort range

WorkerIngressVxlan

Vxlan packets

udp

4789

WorkerIngressWorkerVxlan

Vxlan packets

udp

4789

WorkerIngressInternal

Internal cluster communication

tcp

9000 - 9999

WorkerIngressWorkerInternal

Internal cluster communication

tcp

9000 - 9999

WorkerIngressKube

Kubernetes kubelet, scheduler, and controller manager

tcp

10250

WorkerIngressWorkerKube

Kubernetes kubelet, scheduler, and controller manager

tcp

10250

WorkerIngressIngressServices

Kubernetes Ingress services

tcp

30000 - 32767

WorkerIngressWorkerIngressServices

Kubernetes Ingress services

tcp

30000 - 32767

Roles and instance profiles

You must grant the machines permissions in AWS. The provided CloudFormation templates grant the machines permission the following AWS::IAM::Role objects and provide a AWS::IAM::InstanceProfile for each set of roles. If you do not use the templates, you can grant the machines the following broad permissions or the following individual permissions.

RoleEffectActionResource

Master

Allow

ec2:*

*

Allow

elasticloadbalancing:*

*

Allow

iam:PassRole

*

Allow

s3:GetObject

*

Worker

Allow

ec2:Describe*

*

Bootstrap

Allow

ec2:Describe*

*

Allow

ec2:AttachVolume

*

Allow

ec2:DetachVolume

*