Quay health checks failing when Quay is configured behind an AWS load balancer

Solution Verified - Updated -

Issue

When health check is set on the AWS ELB for Quay, Quay's /health/instance endpoint fails with the following error:

curl -k -s https://localhost/health/instance
{"data":{"services":{"auth":true,"database":true,"disk_space":true,"registry_gunicorn":false,"service_key":true,"verbs_gunicorn":false,"web_gunicorn":false}},"status_code":503}

The following Python trace can be seen in the logs:

2019-08-30 10:13:25,487 [527] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 301 185
2019-08-30 10:13:25,488 [527] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (3): localhost
2019-08-30 10:13:25,491 [527] [ERROR] [health.services] Exception when checking worker health: http://localhost:8080/_internal_ping
Traceback (most recent call last):
  File "/quay-registry/health/services.py", line 36, in fn
    status_code = client.get(registry_url, verify=False, timeout=2).status_code
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/requests/sessions.py", line 640, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/requests/sessions.py", line 218, in resolve_redirects
    **adapter_kwargs
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/requests/adapters.py", line 508, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='localhost', port=443): Max retries exceeded with url: /_internal_ping (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff260d36e10>: Failed to establish a new connection: [Errno 111] Connection refused',))
gunicorn-web stdout | 2019-08-30 10:13:25,487 [527] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 301 185

Environment

Red Hat Quay v3.0.x and above

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In