InvalidIdentityToken when installing OpenShift on AWS using manual authentication mode with STS

Solution Unverified - Updated -

Issue

  • When installing OpenShift on AWS using manual authentication mode with STS, many operators that require credentials with AWS are failing to authenticate to AWS.
  • The Machine Controller logs is reporting InvalidIdentityToken and WebIdentityErr when AssumeRoleWithWebIdentity
$ oc  logs machine-api-controllers-[redacted] -n openshift-machine-api -c machine-controller
(...)
I0704 20:55:07.356052       1 controller.go:175] my-cluster-qntmh-master-0: reconciling Machine
I0704 20:55:07.356106       1 actuator.go:104] my-cluster-qntmh-master-0: actuator checking if machine exists
E0704 20:55:07.782783       1 reconciler.go:266] my-cluster-qntmh-master-0: error getting existing instances: WebIdentityErr: failed to retrieve credentials
caused by: InvalidIdentityToken: Couldn't retrieve verification key from your identity provider,  please reference AssumeRoleWithWebIdentity documentation for requirements
    status code: 400, request id: [redacted]
E0704 20:55:07.782827       1 controller.go:303] my-cluster-qntmh-master-0: failed to check if machine exists: WebIdentityErr: failed to retrieve credentials
caused by: InvalidIdentityToken: Couldn't retrieve verification key from your identity provider,  please reference AssumeRoleWithWebIdentity documentation for requirements
    status code: 400, request id: [redacted]
  • The Image Registry Operator is reporting InvalidIdentityToken and WebIdentityErr when AssumeRoleWithWebIdentity
$ oc logs pod/cluster-image-registry-operator-[redacted] -n openshift-image-registry
(...)
I0704 21:18:07.603948       1 generator.go:60] object *v1.ClusterOperator, Name=image-registry updated: changed:metadata.resourceVersion={"56299" -> "56307"}, changed:status.conditions.1.message={"Progressing: Unable to apply resources: unable to sync storage configuration: WebIdentityErr: failed to retrieve credentials\nProgressing: caused by: InvalidIdentityToken: Couldn't retrieve verification key from your identity provider,  please reference AssumeRoleWithWebIdentity documentation for requirements\nProgressing: \tstatus code: 400, request id: [redacted]" -> "Progressing: Unable to apply resources: unable to sync storage configuration: WebIdentityErr: failed to retrieve credentials\nProgressing: caused by: InvalidIdentityToken: Couldn't retrieve verification key from your identity provider,  please reference AssumeRoleWithWebIdentity documentation for requirements\nProgressing: \tstatus code: 400, request id: [redacted]"}

Environment

  • Red Hat OpenShift Container Platform [RHOCP]
    • 4.8+
  • Installing the cluster with manual authentication mode in AWS with STS
  • AWS Account installing RHOCP does not allow public S3 Buckets or objects

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content