Why does AAP on OpenShift show "Error connecting to Controller API" after an AAP Operator update when a cluster-wide proxy is configured?

Solution Verified - Updated -

Issue

After the Ansible Automation Platform (AAP) Operator updates to version 2.6.0-0.1783986130 (AAP 2.6) or 2.7.0+0.1783928748 (AAP 2.7) on an OpenShift cluster that has a cluster-wide egress proxy configured, the platform stops working even though no configuration was changed by the administrator. Environments without a cluster-wide proxy are not affected.

  • The AAP UI shows a banner after login:
Error connecting to Controller API

or

HTTP 401: Error connecting to Controller API. This may indicate a misconfigured JWT key or service key between the Gateway and Controller. Check Controller logs for details.
  • The automation controller web container logs show the JWT decryption key fetch failing with a 5XX returned by the proxy server:
INFO Loading decryption key from http://<aap-instance-name> scheme http
ERROR Failed to get 200 response from the issuer: 502
INFO status 401 received by user AnonymousUser attempting to access /api/controller/v2/me/
  • The automation hub API and content pods may go NotReady or crash loop, with the readiness probe reaching the proxy instead of the local process:
Readiness probe failed: Readiness probe checking http://[::1]:24816/pulp/content/
...
http.client.BadStatusLine: <HEAD><TITLE>Authentication Required</TITLE></HEAD>
...
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine('<HEAD><TITLE>Authentication Required</TITLE></HEAD>\n'))

Note: the "Error connecting to Controller API" banner has several unrelated causes. This solution applies only when the update trigger, the cluster-wide proxy, and the log signatures above are all present. For other causes of the same banner, see KCS 7145091.

Environment

  • Red Hat Ansible Automation Platform (AAP) 2.6 or 2.7 on Red Hat OpenShift Container Platform
  • Installed with the AAP Operator, version 2.6.0-0.1783986130, 2.7.0+0.1783928748, or later
  • A cluster-wide egress proxy is configured (Proxy object named cluster), and its noProxy list does not include the AAP instance's short service names or the IPv6 loopback address ::1

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content