Red Hat Openshift Online - [ALERT] Proxy express has no server available

Solution In Progress - Updated -

Environment

  • Red Hat OpenShift Online

Issue

  • My servers are down and I am unable to bring them up:
[ALERT] 005/015432 (350193) : proxy 'express' has no server available!
[WARNING] 005/020435 (350193) : Server express/local-gear is DOWN for maintenance.
[WARNING] 005/020435 (472765) : config : log format ignored for proxy 'stats' since it has no log address.
[WARNING] 005/020435 (472765) : config : log format ignored for proxy 'express' since it has no log address.
[WARNING] 005/020435 (472765) : Server express/local-gear is DOWN, reason: Layer7 wrong status, code: 404, info: "Not Found", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 005/020435 (472765) : proxy 'express' has no server available!
[WARNING] 005/021114 (472765) : Server express/local-gear is DOWN for maintenance.
[WARNING] 005/021124 (472765) : Server express/local-gear is UP (leaving maintenance).
[WARNING] 005/021130 (472765) : Server express/local-gear is DOWN, reason: Layer7 wrong status, code: 404, info: "Not Found", check duration: 4875ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 005/021130 (472765) : proxy 'express' has no server available!
  • Cannot access or login to my high availability application
  • Gear is stuck down for maintenance

Resolution

Ensure that the HAProxy is able to perform health checks on the application:

  • ssh into the main gear of the application.
$ rhc ssh [appname]
  • Update the haproxy conf: change the httpchk GET option from / to a valid location, such as an index.html or index.jsp file
$ cd ~/haproxy/conf
$ vim haproxy.cfg
. . .
option httpchk GET /path/to/viable/resource
. . .
  • Restart the HAProxy
$ rhc cartridge-restart --cartridge haproxy
  • Verify the status of your gears by going to http://-.rhcloud.com/haproxy-status

Root Cause

Applications with scaling enabled utilize a High Availability Proxy (HAProxy) to perform health checks and other maintenance related to scaling. The health check URL identified in haproxy.cfg is contacted via the loopback IP on the gear. If the health check URL is inaccessible or otherwise non-valid (does not return a 200 status message on a curl over the loopback interface), the gear will fail the health check.

Diagnostic Steps

  1. Verify the status of your gears by going to http://-.rhcloud.com/haproxy-status
  2. Investigate the location pointed to by the httpchk GET option in the haproxy/conf/haproxy.cfg file
$ curl <location indicated by the httpchk GET option>

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments