ROSA installation fails when using cluster-wide proxy

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Service on AWS (ROSA)
    • 4

Issue

  • When installing a ROSA cluster using a cluster-wide proxy, the installation even if the prerequisites for configuring a cluster-wide proxy are met.
  • The installation of a ROSA cluster fail with message OCM3999 Unknown error. Check this cluster's installation logs for more details, or delete this cluster and try again. If this issue persists, contact support when a cluster-wide proxy is used for the installation.

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

When installing ROSA using a cluster-wide proxy the endpoints must be configured as type Gateway, otherwise the installation will fail.

The endpoints needed for installation are noted in the documentation in the General requirements section at:

Three endpoints are required in the VPC:

  • ec2.[aws_region].amazonaws.com
  • elasticloadbalancing.[aws_region].amazonaws.com
  • s3.[aws_region].amazonaws.com

More details at AWS Knowledge base for s3 endpoint configuration at:

Root Cause

Because the proxy works on the container level, and not the node level, it is needed to route these requests to the AWS EC2 API through the AWS Private Network. Adding the public IP address of the EC2 API to the allowlist in the proxy server is not sufficient.

However, the endpoint must be of type Gateway, as if an endpoint for S3 is created as type Interface, the installation will fail.

Diagnostic Steps

Configure the AWS pre-requisites and the cluster-wide proxy prerequisites as per the documentation at the following URL: Configure Cluster-wide Proxy.

  • Create the endpoints of type Interface, and the other prereqisistes.
  • Run the ROSA installer with the cluster-wide configuration.
  • Control Plane API never comes up (if non-STS, it is possible to see ignition configs are not accessible in s3, for STS clusters, no logs are available at provision time unless remounting the storage for the bootstrap and master nodes on another VM in the same VPC).

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments