Quay health checks failing when Quay is configured behind an AWS load balancer
Issue
When health check is set on the AWS ELB for Quay, Quay's /health/instance
endpoint fails with the following error:
curl -k -s https://localhost/health/instance
{"data":{"services":{"auth":true,"database":true,"disk_space":true,"registry_gunicorn":false,"service_key":true,"verbs_gunicorn":false,"web_gunicorn":false}},"status_code":503}
The following Python trace can be seen in the logs:
2019-08-30 10:13:25,487 [527] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 301 185
2019-08-30 10:13:25,488 [527] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (3): localhost
2019-08-30 10:13:25,491 [527] [ERROR] [health.services] Exception when checking worker health: http://localhost:8080/_internal_ping
Traceback (most recent call last):
File "/quay-registry/health/services.py", line 36, in fn
status_code = client.get(registry_url, verify=False, timeout=2).status_code
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/requests/sessions.py", line 521, in get
return self.request('GET', url, **kwargs)
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/requests/sessions.py", line 640, in send
history = [resp for resp in gen] if allow_redirects else []
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/requests/sessions.py", line 218, in resolve_redirects
**adapter_kwargs
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='localhost', port=443): Max retries exceeded with url: /_internal_ping (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff260d36e10>: Failed to establish a new connection: [Errno 111] Connection refused',))
gunicorn-web stdout | 2019-08-30 10:13:25,487 [527] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 301 185
Environment
Red Hat Quay v3.0.x and above
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.