Provisioning ROSA Cluster with S3 Bucket Versioning Enabled

Solution Verified - Updated -

Environment

ROSA w/ S3 Bucket Versioning enabled in the provision account.

Issue

As per AWS best practices, many customers may want to have S3 bucket versioning enabled and may enforce this practice (see https://docs.aws.amazon.com/AmazonS3/latest/userguide/security-best-practices.html#security-best-practices-prevent). However, provisioning a cluster with S3 bucket versioning fails. This is because the installer is missing required permissions that have not been propagated to the Installer role.

Resolution

Create a new AWS IAM policy with the following permissions:

  • s3:DeleteObjectVersion
  • s3:PutBucketVersioning

Attach the newly created IAM policy to the Installer IAM role (e.g. ManagedOpenShift-Installer-Role)

To create the policies and attach them with the AWS CLI (be sure to substitute the appropriate names):

ROLE_NAME=ManagedOpenshift-Installer-Role
POLICY_NAME=rosa-s3-policy-fix
POLICY_ARN=$(aws iam create-policy --policy-name $POLICY_NAME --policy-document '{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": ["s3:DeleteObjectVersion", "s3:PutBucketVersioning"], "Resource": "*"}]}')

aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn $POLICY_ARN

Root Cause

The installer role is missing the required permissions to enable the workflow to provision a ROSA cluster when S3 bucket versioning is enabled.

Diagnostic Steps

While experiencing a failed ROSA cluster provision, you may see ROSA installer logs, either in the console or from the rosa logs install command with similar output to the following:

level=info msg=Waiting up to 30m0s (until 1:58PM) for bootstrapping to complete...
level=info msg=Destroying the bootstrap resources...
level=error
level=error msg=Error: error deleting S3 Bucket (cluster-bootstrap) Object (bootstrap.ign): error deleting at least one object version, last error: AccessDenied: Access Denied
level=error msg=       status code: 403, request id: FBCQ93P8Q2TRS5HA, host id: WKe6l8/RysxBE/UVqmooegDWg3rNopdOcbZTo2q9Bi75d2A+f9LEXGU4YINPm1SVh27gMnaIc24=
level=error
level=fatal msg=terraform destroy: failed doing terraform destroy: exit status 1
level=fatal
level=fatal msg=Error: error deleting S3 Bucket (cluster-bootstrap) Object (bootstrap.ign): error deleting at least one object version, last error: AccessDenied: Access Denied
level=fatal msg=       status code: 403, request id: FBCQ93P8Q2TRS5HA, host id: WKe6l8/RysxBE/UVqmooegDWg3rNopdOcbZTo2q9Bi75d2A+f9LEXGU4YINPm1SVh27gMnaIc24=
level=fatal
level=fatal
" installID=9k8pjbnx

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments