OpenShift 4.x installs may fail on various AWS region due to AWS S3 policy change

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP) 4.11, 4.12
    • Installer-Provisioned Infrastructure (IPI) on AWS
  • Red Hat OpenShift Service on AWS (ROSA) 4.11, 4.12
  • OpenShift Dedicated (OSD) 4.11, 4.12
  • Amazon Web Services (AWS) Cloud Regions

Issue

  • OpenShift 4 installation on AWS using IPI is failing due to AWS S3 policy change.
  • OpenShift 4 installation on AWS is failing with below error reported.
  • Red Hat OpenShift Services on AWS (ROSA) is failing with below error reported.

  • OpenShift Dedicated (OSD) is failing due to AWS S3 policy change and is failing with below error reported.

    time="2023-04-11T12:49:19Z" level=error msg="Error: error creating S3 bucket ACL for foo1234-r8l8v-bootstrap: AccessControlListNotSupported: The bucket does not allow ACLs"
time="2023-04-11T12:49:19Z" level=error msg="\tstatus code: 400, request id: MSFHN7834MSD, host id: osdhfohsjdfnohunwejsfhusjfniusdfidsfhuishfuisdbf"

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

  • This problem was resolved in Red Hat OpenShift Container Platform 4.11.36 via RHBA-2023:1733 and in Red Hat OpenShift Container Platform 4.12.12 via RHBA-2023:1734.
    • As AWS rolls out the S3 Security changes to all regions, only Red Hat OpenShift Container Platform 4.11.36 or 4.12.12 and later shall be used for Red Hat OpenShift Container Platform 4 installation on AWS, using Installer-Provisioned Infrastructure method as older version will fail with the error documentation in this solution.

Technical Details

  • Only new Red Hat OpenShift Container Platform 4.11 and 4.12 Cluster installation on AWS, using Installer-Provisioned Infrastructure (IPI), ROSA or OSD installation method are affected.
    • Running Red Hat OpenShift Container Platform 4 - Clusters are not impacted and also updates will continue to work.
  • Only Red Hat OpenShift Container Platform 4.11 and 4.12 are affected by this issue. Red Hat OpenShift Container Platform 4.10 and earlier are NOT affected and will continue to install successfully when all prerequisites are met.
  • Red Hat had temporarily disabled the creation of Red Hat OpenShift Service on AWS (ROSA) and OpenShift Dedicated (OSD) clusters in the AWS us-east-2 region until the resolution was released. This is no longer disabled.

Root Cause

The problem is related to the rollout of the Amazon S3 Security Changes that started on April 11th 2023 in us-east-2 region.

Diagnostic Steps

  • Red Hat OpenShift Container Platform 4 installation on AWS will report errors as shown below.

    time="2023-04-11T12:49:19Z" level=debug msg="aws_lb_target_group_attachment.bootstrap[2]: Creation complete after 1s [id=arn:aws:elasticloadbalancing:us-east-2:123456789:targetgroup/foo1234abc-r8l8v-aext/1234-5678]"
time="2023-04-11T12:49:19Z" level=debug msg="aws_lb_target_group_attachment.bootstrap[0]: Creation complete after 1s [id=arn:aws:elasticloadbalancing:us-east-2:123456789:targetgroup/foo1234abc-r8l8v-aint/4567-9876]"
time="2023-04-11T12:49:19Z" level=debug msg="aws_lb_target_group_attachment.bootstrap[1]: Creation complete after 1s [id=arn:aws:elasticloadbalancing:us-east-2:123456789:targetgroup/foo1234abc-r8l8v-sint/9876-1234]"
time="2023-04-11T12:49:19Z" level=debug
time="2023-04-11T12:49:19Z" level=debug msg="Warning: Value for undeclared variable"
time="2023-04-11T12:49:19Z" level=debug
time="2023-04-11T12:49:19Z" level=debug msg="The root module does not declare a variable named \"control_plane_ips\" but a"
time="2023-04-11T12:49:19Z" level=debug msg="value was found in file"
time="2023-04-11T12:49:19Z" level=debug msg="\"/tmp/openshift-install-bootstrap-2907866028/cluster.tfvars.json\". If you"
time="2023-04-11T12:49:19Z" level=debug msg="meant to use this value, add a \"variable\" block to the configuration."
time="2023-04-11T12:49:19Z" level=debug
time="2023-04-11T12:49:19Z" level=debug msg="To silence these warnings, use TF_VAR_... environment variables to provide"
time="2023-04-11T12:49:19Z" level=debug msg="certain \"global\" settings to all configurations in your organization. To"
time="2023-04-11T12:49:19Z" level=debug msg="reduce the verbosity of these warnings, use the -compact-warnings option."
time="2023-04-11T12:49:19Z" level=error
time="2023-04-11T12:49:19Z" level=error msg="Error: error creating S3 bucket ACL for foo1234abc-r8l8v-bootstrap: AccessControlListNotSupported: The bucket does not allow ACLs"
time="2023-04-11T12:49:19Z" level=error msg="\tstatus code: 400, request id: SDFIJHDIOFNK34234, host id: soisdhodsndsfuihuihewr32z4894z8zuhfsuhfu=="
time="2023-04-11T12:49:19Z" level=error
time="2023-04-11T12:49:19Z" level=error msg="  with aws_s3_bucket_acl.ignition,"
time="2023-04-11T12:49:19Z" level=error msg="  on main.tf line 62, in resource \"aws_s3_bucket_acl\" \"ignition\":"
time="2023-04-11T12:49:19Z" level=error msg="  62: resource \"aws_s3_bucket_acl\" ignition {"
time="2023-04-11T12:49:19Z" level=error
time="2023-04-11T12:49:19Z" level=error msg="failed to fetch Cluster: failed to generate asset \"Cluster\": failure applying terraform for \"bootstrap\" stage: failed to create cluster: failed to apply Terraform: exit status 1\n\nError: error creating S3 bucket ACL for foo1234abc-r8l8v-bootstrap: AccessControlListNotSupported: The bucket does not allow ACLs\n\tstatus code: 400, request id: SDFIJHDIOFNK34234, host id: soisdhodsndsfuihuihewr32z4894z8zuhfsuhfu==\n\n  with aws_s3_bucket_acl.ignition,\n  on main.tf line 62, in resource \"aws_s3_bucket_acl\" \"ignition\":\n  62: resource \"aws_s3_bucket_acl\" ignition {\n\n"
time="2023-04-11T12:49:20Z" level=error msg="error after waiting for command completion" error="exit status 4" installID=2ssfbfvm
time="2023-04-11T12:49:20Z" level=error msg="error provisioning cluster" error="exit status 4" installID=2ssfbfvm

  • For Red Hat OpenShift Services on AWS(ROSA) and OpenShift Dedicated(OSD) the installation will fail.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments