4.11.13. 在 AWS 中创建 control plane 机器

您必须在集群要使用的 Amazon Web Services(AWS)中创建 control plane 机器。

您可以使用提供的 CloudFormation 模板和自定义参数文件,创建代表 control plane 节点的 AWS 资源堆栈。

重要

CloudFormation 模板会创建一个堆栈,它代表三个 control plane 节点。

注意

如果不使用提供的 CloudFormation 模板来创建 control plane 节点,您必须检查提供的信息并手动创建基础架构。如果集群没有正确初始化,您可能需要联系红帽支持并提供您的安装日志。

先决条件

  • 已配置了一个 AWS 帐户。
  • 您可以通过运行 aws configure,将 AWS 密钥和区域添加到本地 AWS 配置集中。
  • 已为集群生成 Ignition 配置文件。
  • 您在 AWS 中创建并配置了 VPC 及相关子网。
  • 您在 AWS 中创建并配置了 DNS、负载均衡器和监听程序。
  • 您在 AWS 中创建了集群所需的安全组和角色。
  • 已创建 bootstrap 机器。

流程

  1. 创建一个 JSON 文件,其包含模板所需的参数值:

    [
      {
        "ParameterKey": "InfrastructureName", 1
        "ParameterValue": "mycluster-<random_string>" 2
      },
      {
        "ParameterKey": "RhcosAmi", 3
        "ParameterValue": "ami-<random_string>" 4
      },
      {
        "ParameterKey": "AutoRegisterDNS", 5
        "ParameterValue": "yes" 6
      },
      {
        "ParameterKey": "PrivateHostedZoneId", 7
        "ParameterValue": "<random_string>" 8
      },
      {
        "ParameterKey": "PrivateHostedZoneName", 9
        "ParameterValue": "mycluster.example.com" 10
      },
      {
        "ParameterKey": "Master0Subnet", 11
        "ParameterValue": "subnet-<random_string>" 12
      },
      {
        "ParameterKey": "Master1Subnet", 13
        "ParameterValue": "subnet-<random_string>" 14
      },
      {
        "ParameterKey": "Master2Subnet", 15
        "ParameterValue": "subnet-<random_string>" 16
      },
      {
        "ParameterKey": "MasterSecurityGroupId", 17
        "ParameterValue": "sg-<random_string>" 18
      },
      {
        "ParameterKey": "IgnitionLocation", 19
        "ParameterValue": "https://api-int.<cluster_name>.<domain_name>:22623/config/master" 20
      },
      {
        "ParameterKey": "CertificateAuthorities", 21
        "ParameterValue": "data:text/plain;charset=utf-8;base64,ABC...xYz==" 22
      },
      {
        "ParameterKey": "MasterInstanceProfileName", 23
        "ParameterValue": "<roles_stack>-MasterInstanceProfile-<random_string>" 24
      },
      {
        "ParameterKey": "MasterInstanceType", 25
        "ParameterValue": "m5.xlarge" 26
      },
      {
        "ParameterKey": "AutoRegisterELB", 27
        "ParameterValue": "yes" 28
      },
      {
        "ParameterKey": "RegisterNlbIpTargetsLambdaArn", 29
        "ParameterValue": "arn:aws:lambda:<region>:<account_number>:function:<dns_stack_name>-RegisterNlbIpTargets-<random_string>" 30
      },
      {
        "ParameterKey": "ExternalApiTargetGroupArn", 31
        "ParameterValue": "arn:aws:elasticloadbalancing:<region>:<account_number>:targetgroup/<dns_stack_name>-Exter-<random_string>" 32
      },
      {
        "ParameterKey": "InternalApiTargetGroupArn", 33
        "ParameterValue": "arn:aws:elasticloadbalancing:<region>:<account_number>:targetgroup/<dns_stack_name>-Inter-<random_string>" 34
      },
      {
        "ParameterKey": "InternalServiceTargetGroupArn", 35
        "ParameterValue": "arn:aws:elasticloadbalancing:<region>:<account_number>:targetgroup/<dns_stack_name>-Inter-<random_string>" 36
      }
    ]
    1
    您的 Ignition 配置文件中为集群编码的集群基础架构名称。
    2
    指定从 Ignition 配置文件元数据中提取的基础架构名称,其格式为 <cluster-name>-<random-string>
    3
    用于 control plane 机器的当前 Red Hat Enterprise Linux CoreOS (RHCOS) AMI。
    4
    指定 AWS::EC2::Image::Id 值。
    5
    是否要执行 DNS etcd 注册。
    6
    指定 yesno。如果指定 yes,您必须提供托管区信息。
    7
    用来注册 etcd 目标的 Route 53 专用区 ID。
    8
    指定 DNS 和负载均衡的 CloudFormation 模板输出的 PrivateHostedZoneId 值。
    9
    用来注册目标的 Route 53 区。
    10
    指定 <cluster_name>.<domain_name>,其中 <domain_name> 是您为集群生成 install-config.yaml 文件时所用的 Route 53 基域。请勿包含 AWS 控制台中显示的结尾句点 (.)。
    11 13 15
    在其中启动 control plane 机器的子网,最好是专用子网。
    12 14 16
    从 DNS 和负载均衡的 CloudFormation 模板输出的 PrivateSubnets 值指定子网。
    17
    与 control plane 节点(也称为 master 节点)关联的 master 安全组 ID。
    18
    指定安全组和角色的 CloudFormation 模板输出的 MasterSecurityGroupId 值。
    19
    从中获取 control plane Ignition 配置文件的位置。
    20
    指定生成的 Ignition 配置文件的位置,https://api-int.<cluster_name>.<domain_name>:22623/config/master
    21
    要使用的 base64 编码证书颁发机构字符串。
    22
    指定安装目录中 master.ign 文件中的值。这个值是一个长字符串,格式为 data:text/plain;charset=utf-8;base64,ABC…​xYz==
    23
    与 control plane 节点关联的 IAM 配置集。
    24
    指定安全组和角色的 CloudFormation 模板输出的 MasterInstanceProfile 参数值。
    25
    用于 control plane 机器的 AWS 实例类型。
    26
    允许的值:
    • m4.xlarge
    • m4.2xlarge
    • m4.4xlarge
    • m4.10xlarge
    • m4.16xlarge
    • m5.xlarge
    • m5.2xlarge
    • m5.4xlarge
    • m5.8xlarge
    • m5.12xlarge
    • m5.16xlarge
    • m5a.xlarge
    • m5a.2xlarge
    • m5a.4xlarge
    • m5a.8xlarge
    • m5a.10xlarge
    • m5a.16xlarge
    • c4.2xlarge
    • c4.4xlarge
    • c4.8xlarge
    • c5.2xlarge
    • c5.4xlarge
    • c5.9xlarge
    • c5.12xlarge
    • c5.18xlarge
    • c5.24xlarge
    • c5a.2xlarge
    • c5a.4xlarge
    • c5a.8xlarge
    • c5a.12xlarge
    • c5a.16xlarge
    • c5a.24xlarge
    • r4.xlarge
    • r4.2xlarge
    • r4.4xlarge
    • r4.8xlarge
    • r4.16xlarge
    • r5.xlarge
    • r5.2xlarge
    • r5.4xlarge
    • r5.8xlarge
    • r5.12xlarge
    • r5.16xlarge
    • r5.24xlarge
    • r5a.xlarge
    • r5a.2xlarge
    • r5a.4xlarge
    • r5a.8xlarge
    • r5a.12xlarge
    • r5a.16xlarge
    • r5a.24xlarge
    27
    是否要注册网络负载均衡器 (NLB) 。
    28
    指定 yesno。如果指定 yes,您必须提供一个 Lambda Amazon Resource Name (ARN) 值。
    29
    NLB IP 目标注册 lambda 组的 ARN。
    30
    指定 DNS 和负载均衡的 CloudFormation 模板输出的 RegisterNlbIpTargetsLambda 值。如果将集群部署到 AWS GovCloud 区域,请使用 arn:aws-us-gov
    31
    外部 API 负载均衡器目标组的 ARN。
    32
    指定 DNS 和负载均衡的 CloudFormation 模板输出的 ExternalApiTargetGroupArn 值。如果将集群部署到 AWS GovCloud 区域,请使用 arn:aws-us-gov
    33
    内部 API 负载均衡器目标组群的 ARN。
    34
    指定 DNS 和负载均衡的 CloudFormation 模板输出的 InternalApiTargetGroupArn 值。如果将集群部署到 AWS GovCloud 区域,请使用 arn:aws-us-gov
    35
    内部服务负载均衡器目标组群的 ARN。
    36
    指定 DNS 和负载均衡的 CloudFormation 模板输出的 InternalServiceTargetGroupArn 值。如果将集群部署到 AWS GovCloud 区域,请使用 arn:aws-us-gov
  2. 复制control plane 机器的 CloudFormation 模板一节中的模板,并将它以 YAML 文件形式保存到计算机上。此模板描述了集群所需的 control plane 机器。
  3. 如果您将 m5 实例类型指定为 MasterInstanceType 的值,请将该实例类型添加到 CloudFormation 模板中的 MasterInstanceType.AllowedValues 参数。
  4. 启动 CloudFormation 模板,以创建代表 control plane 节点的 AWS 资源堆栈:

    重要

    您必须在一行内输入命令。

    $ aws cloudformation create-stack --stack-name <name> 1
         --template-body file://<template>.yaml 2
         --parameters file://<parameters>.json 3
    1
    <name> 是 CloudFormation 堆栈的名称,如 cluster-control-plane。如果您删除集群,则需要此堆栈的名称。
    2
    <template> 是您保存的 CloudFormation 模板 YAML 文件的相对路径和名称。
    3
    <parameters> 是 CloudFormation 参数 JSON 文件的相对路径和名称。

    输出示例

    arn:aws:cloudformation:us-east-1:269333783861:stack/cluster-control-plane/21c7e2b0-2ee2-11eb-c6f6-0aa34627df4b

    注意

    CloudFormation 模板会创建一个堆栈,它代表三个 control plane 节点。

  5. 确认模板组件已存在:

    $ aws cloudformation describe-stacks --stack-name <name>

4.11.13.1. control plane 机器的 CloudFormation 模板

您可以使用以下 CloudFormation 模板来部署 OpenShift Container Platform 集群所需的 control plane 机器。

例 4.39. control plane 机器的 CloudFormation 模板

AWSTemplateFormatVersion: 2010-09-09
Description: Template for OpenShift Cluster Node Launch (EC2 master instances)

Parameters:
  InfrastructureName:
    AllowedPattern: ^([a-zA-Z][a-zA-Z0-9\-]{0,26})$
    MaxLength: 27
    MinLength: 1
    ConstraintDescription: Infrastructure name must be alphanumeric, start with a letter, and have a maximum of 27 characters.
    Description: A short, unique cluster ID used to tag nodes for the kubelet cloud provider.
    Type: String
  RhcosAmi:
    Description: Current Red Hat Enterprise Linux CoreOS AMI to use for bootstrap.
    Type: AWS::EC2::Image::Id
  AutoRegisterDNS:
    Default: "yes"
    AllowedValues:
    - "yes"
    - "no"
    Description: Do you want to invoke DNS etcd registration, which requires Hosted Zone information?
    Type: String
  PrivateHostedZoneId:
    Description: The Route53 private zone ID to register the etcd targets with, such as Z21IXYZABCZ2A4.
    Type: String
  PrivateHostedZoneName:
    Description: The Route53 zone to register the targets with, such as cluster.example.com. Omit the trailing period.
    Type: String
  Master0Subnet:
    Description: The subnets, recommend private, to launch the master nodes into.
    Type: AWS::EC2::Subnet::Id
  Master1Subnet:
    Description: The subnets, recommend private, to launch the master nodes into.
    Type: AWS::EC2::Subnet::Id
  Master2Subnet:
    Description: The subnets, recommend private, to launch the master nodes into.
    Type: AWS::EC2::Subnet::Id
  MasterSecurityGroupId:
    Description: The master security group ID to associate with master nodes.
    Type: AWS::EC2::SecurityGroup::Id
  IgnitionLocation:
    Default: https://api-int.$CLUSTER_NAME.$DOMAIN:22623/config/master
    Description: Ignition config file location.
    Type: String
  CertificateAuthorities:
    Default: data:text/plain;charset=utf-8;base64,ABC...xYz==
    Description: Base64 encoded certificate authority string to use.
    Type: String
  MasterInstanceProfileName:
    Description: IAM profile to associate with master nodes.
    Type: String
  MasterInstanceType:
    Default: m5.xlarge
    Type: String
    AllowedValues:
    - "m4.xlarge"
    - "m4.2xlarge"
    - "m4.4xlarge"
    - "m4.10xlarge"
    - "m4.16xlarge"
    - "m5.xlarge"
    - "m5.2xlarge"
    - "m5.4xlarge"
    - "m5.8xlarge"
    - "m5.12xlarge"
    - "m5.16xlarge"
    - "m5a.xlarge"
    - "m5a.2xlarge"
    - "m5a.4xlarge"
    - "m5a.8xlarge"
    - "m5a.10xlarge"
    - "m5a.16xlarge"
    - "c4.2xlarge"
    - "c4.4xlarge"
    - "c4.8xlarge"
    - "c5.2xlarge"
    - "c5.4xlarge"
    - "c5.9xlarge"
    - "c5.12xlarge"
    - "c5.18xlarge"
    - "c5.24xlarge"
    - "c5a.2xlarge"
    - "c5a.4xlarge"
    - "c5a.8xlarge"
    - "c5a.12xlarge"
    - "c5a.16xlarge"
    - "c5a.24xlarge"
    - "r4.xlarge"
    - "r4.2xlarge"
    - "r4.4xlarge"
    - "r4.8xlarge"
    - "r4.16xlarge"
    - "r5.xlarge"
    - "r5.2xlarge"
    - "r5.4xlarge"
    - "r5.8xlarge"
    - "r5.12xlarge"
    - "r5.16xlarge"
    - "r5.24xlarge"
    - "r5a.xlarge"
    - "r5a.2xlarge"
    - "r5a.4xlarge"
    - "r5a.8xlarge"
    - "r5a.12xlarge"
    - "r5a.16xlarge"
    - "r5a.24xlarge"

  AutoRegisterELB:
    Default: "yes"
    AllowedValues:
    - "yes"
    - "no"
    Description: Do you want to invoke NLB registration, which requires a Lambda ARN parameter?
    Type: String
  RegisterNlbIpTargetsLambdaArn:
    Description: ARN for NLB IP target registration lambda. Supply the value from the cluster infrastructure or select "no" for AutoRegisterELB.
    Type: String
  ExternalApiTargetGroupArn:
    Description: ARN for external API load balancer target group. Supply the value from the cluster infrastructure or select "no" for AutoRegisterELB.
    Type: String
  InternalApiTargetGroupArn:
    Description: ARN for internal API load balancer target group. Supply the value from the cluster infrastructure or select "no" for AutoRegisterELB.
    Type: String
  InternalServiceTargetGroupArn:
    Description: ARN for internal service load balancer target group. Supply the value from the cluster infrastructure or select "no" for AutoRegisterELB.
    Type: String

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
    - Label:
        default: "Cluster Information"
      Parameters:
      - InfrastructureName
    - Label:
        default: "Host Information"
      Parameters:
      - MasterInstanceType
      - RhcosAmi
      - IgnitionLocation
      - CertificateAuthorities
      - MasterSecurityGroupId
      - MasterInstanceProfileName
    - Label:
        default: "Network Configuration"
      Parameters:
      - VpcId
      - AllowedBootstrapSshCidr
      - Master0Subnet
      - Master1Subnet
      - Master2Subnet
    - Label:
        default: "DNS"
      Parameters:
      - AutoRegisterDNS
      - PrivateHostedZoneName
      - PrivateHostedZoneId
    - Label:
        default: "Load Balancer Automation"
      Parameters:
      - AutoRegisterELB
      - RegisterNlbIpTargetsLambdaArn
      - ExternalApiTargetGroupArn
      - InternalApiTargetGroupArn
      - InternalServiceTargetGroupArn
    ParameterLabels:
      InfrastructureName:
        default: "Infrastructure Name"
      VpcId:
        default: "VPC ID"
      Master0Subnet:
        default: "Master-0 Subnet"
      Master1Subnet:
        default: "Master-1 Subnet"
      Master2Subnet:
        default: "Master-2 Subnet"
      MasterInstanceType:
        default: "Master Instance Type"
      MasterInstanceProfileName:
        default: "Master Instance Profile Name"
      RhcosAmi:
        default: "Red Hat Enterprise Linux CoreOS AMI ID"
      BootstrapIgnitionLocation:
        default: "Master Ignition Source"
      CertificateAuthorities:
        default: "Ignition CA String"
      MasterSecurityGroupId:
        default: "Master Security Group ID"
      AutoRegisterDNS:
        default: "Use Provided DNS Automation"
      AutoRegisterELB:
        default: "Use Provided ELB Automation"
      PrivateHostedZoneName:
        default: "Private Hosted Zone Name"
      PrivateHostedZoneId:
        default: "Private Hosted Zone ID"

Conditions:
  DoRegistration: !Equals ["yes", !Ref AutoRegisterELB]
  DoDns: !Equals ["yes", !Ref AutoRegisterDNS]

Resources:
  Master0:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref RhcosAmi
      BlockDeviceMappings:
      - DeviceName: /dev/xvda
        Ebs:
          VolumeSize: "120"
          VolumeType: "gp2"
      IamInstanceProfile: !Ref MasterInstanceProfileName
      InstanceType: !Ref MasterInstanceType
      NetworkInterfaces:
      - AssociatePublicIpAddress: "false"
        DeviceIndex: "0"
        GroupSet:
        - !Ref "MasterSecurityGroupId"
        SubnetId: !Ref "Master0Subnet"
      UserData:
        Fn::Base64: !Sub
        - '{"ignition":{"config":{"merge":[{"source":"${SOURCE}"}]},"security":{"tls":{"certificateAuthorities":[{"source":"${CA_BUNDLE}"}]}},"version":"3.1.0"}}'
        - {
          SOURCE: !Ref IgnitionLocation,
          CA_BUNDLE: !Ref CertificateAuthorities,
        }
      Tags:
      - Key: !Join ["", ["kubernetes.io/cluster/", !Ref InfrastructureName]]
        Value: "shared"

  RegisterMaster0:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref ExternalApiTargetGroupArn
      TargetIp: !GetAtt Master0.PrivateIp

  RegisterMaster0InternalApiTarget:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref InternalApiTargetGroupArn
      TargetIp: !GetAtt Master0.PrivateIp

  RegisterMaster0InternalServiceTarget:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref InternalServiceTargetGroupArn
      TargetIp: !GetAtt Master0.PrivateIp

  Master1:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref RhcosAmi
      BlockDeviceMappings:
      - DeviceName: /dev/xvda
        Ebs:
          VolumeSize: "120"
          VolumeType: "gp2"
      IamInstanceProfile: !Ref MasterInstanceProfileName
      InstanceType: !Ref MasterInstanceType
      NetworkInterfaces:
      - AssociatePublicIpAddress: "false"
        DeviceIndex: "0"
        GroupSet:
        - !Ref "MasterSecurityGroupId"
        SubnetId: !Ref "Master1Subnet"
      UserData:
        Fn::Base64: !Sub
        - '{"ignition":{"config":{"merge":[{"source":"${SOURCE}"}]},"security":{"tls":{"certificateAuthorities":[{"source":"${CA_BUNDLE}"}]}},"version":"3.1.0"}}'
        - {
          SOURCE: !Ref IgnitionLocation,
          CA_BUNDLE: !Ref CertificateAuthorities,
        }
      Tags:
      - Key: !Join ["", ["kubernetes.io/cluster/", !Ref InfrastructureName]]
        Value: "shared"

  RegisterMaster1:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref ExternalApiTargetGroupArn
      TargetIp: !GetAtt Master1.PrivateIp

  RegisterMaster1InternalApiTarget:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref InternalApiTargetGroupArn
      TargetIp: !GetAtt Master1.PrivateIp

  RegisterMaster1InternalServiceTarget:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref InternalServiceTargetGroupArn
      TargetIp: !GetAtt Master1.PrivateIp

  Master2:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref RhcosAmi
      BlockDeviceMappings:
      - DeviceName: /dev/xvda
        Ebs:
          VolumeSize: "120"
          VolumeType: "gp2"
      IamInstanceProfile: !Ref MasterInstanceProfileName
      InstanceType: !Ref MasterInstanceType
      NetworkInterfaces:
      - AssociatePublicIpAddress: "false"
        DeviceIndex: "0"
        GroupSet:
        - !Ref "MasterSecurityGroupId"
        SubnetId: !Ref "Master2Subnet"
      UserData:
        Fn::Base64: !Sub
        - '{"ignition":{"config":{"merge":[{"source":"${SOURCE}"}]},"security":{"tls":{"certificateAuthorities":[{"source":"${CA_BUNDLE}"}]}},"version":"3.1.0"}}'
        - {
          SOURCE: !Ref IgnitionLocation,
          CA_BUNDLE: !Ref CertificateAuthorities,
        }
      Tags:
      - Key: !Join ["", ["kubernetes.io/cluster/", !Ref InfrastructureName]]
        Value: "shared"

  RegisterMaster2:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref ExternalApiTargetGroupArn
      TargetIp: !GetAtt Master2.PrivateIp

  RegisterMaster2InternalApiTarget:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref InternalApiTargetGroupArn
      TargetIp: !GetAtt Master2.PrivateIp

  RegisterMaster2InternalServiceTarget:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref InternalServiceTargetGroupArn
      TargetIp: !GetAtt Master2.PrivateIp

  EtcdSrvRecords:
    Condition: DoDns
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !Ref PrivateHostedZoneId
      Name: !Join [".", ["_etcd-server-ssl._tcp", !Ref PrivateHostedZoneName]]
      ResourceRecords:
      - !Join [
        " ",
        ["0 10 2380", !Join [".", ["etcd-0", !Ref PrivateHostedZoneName]]],
      ]
      - !Join [
        " ",
        ["0 10 2380", !Join [".", ["etcd-1", !Ref PrivateHostedZoneName]]],
      ]
      - !Join [
        " ",
        ["0 10 2380", !Join [".", ["etcd-2", !Ref PrivateHostedZoneName]]],
      ]
      TTL: 60
      Type: SRV

  Etcd0Record:
    Condition: DoDns
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !Ref PrivateHostedZoneId
      Name: !Join [".", ["etcd-0", !Ref PrivateHostedZoneName]]
      ResourceRecords:
      - !GetAtt Master0.PrivateIp
      TTL: 60
      Type: A

  Etcd1Record:
    Condition: DoDns
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !Ref PrivateHostedZoneId
      Name: !Join [".", ["etcd-1", !Ref PrivateHostedZoneName]]
      ResourceRecords:
      - !GetAtt Master1.PrivateIp
      TTL: 60
      Type: A

  Etcd2Record:
    Condition: DoDns
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !Ref PrivateHostedZoneId
      Name: !Join [".", ["etcd-2", !Ref PrivateHostedZoneName]]
      ResourceRecords:
      - !GetAtt Master2.PrivateIp
      TTL: 60
      Type: A

Outputs:
  PrivateIPs:
    Description: The control-plane node private IP addresses.
    Value:
      !Join [
        ",",
        [!GetAtt Master0.PrivateIp, !GetAtt Master1.PrivateIp, !GetAtt Master2.PrivateIp]
      ]

其他资源