Quay health checks failing when Quay is configured behind an AWS load balancer

Solution Verified - Updated -

Issue

When health check is set on the AWS ELB for Quay, Quay's /health/instance endpoint fails with the following error:

curl -k -s https://localhost/health/instance
{"data":{"services":{"auth":true,"database":true,"disk_space":true,"registry_gunicorn":false,"service_key":true,"verbs_gunicorn":false,"web_gunicorn":false}},"status_code":503}

The following Python trace can be seen in the logs:

2019-08-30 10:13:25,487 [527] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 301 185
2019-08-30 10:13:25,488 [527] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (3): localhost
2019-08-30 10:13:25,491 [527] [ERROR] [health.services] Exception when checking worker health: http://localhost:8080/_internal_ping
Traceback (most recent call last):
  File "/quay-registry/health/services.py", line 36, in fn
    status_code = client.get(registry_url, verify=False, timeout=2).status_code
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/requests/sessions.py", line 640, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/requests/sessions.py", line 218, in resolve_redirects
    **adapter_kwargs
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/requests/adapters.py", line 508, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='localhost', port=443): Max retries exceeded with url: /_internal_ping (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff260d36e10>: Failed to establish a new connection: [Errno 111] Connection refused',))
gunicorn-web stdout | 2019-08-30 10:13:25,487 [527] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 301 185

Environment

Red Hat Quay v3.0.x and above

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content