OpenShift Router fails with "reload-haproxy: fork: retry: Resource temporarily unavailable" error

Solution Verified - Updated -

Issue

  • On OpenShift Container Platform 4.6, we are seeing the OpenShift Routers crashing with the following error message:

    [..]
    2021-03-08T10:11:25.812338170Z I0308 10:11:25.812241       1 router.go:569] template "msg"="router reloaded"  "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
    2021-03-08T10:11:32.308512149Z I0308 10:11:32.308469       1 router.go:569] template "msg"="router reloaded"  "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
    2021-03-08T10:11:47.451814165Z I0308 10:11:47.451761       1 router.go:569] template "msg"="router reloaded"  "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
    2021-03-08T10:11:56.483946519Z I0308 10:11:56.483892       1 router.go:569] template "msg"="router reloaded"  "output"=" - Proxy protocol on, checking http://localhost:80 ...\n/var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable\n/var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable\n/var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable\n - Health check ok : 0 retry attempt(s).\n"
    2021-03-08T10:12:20.364596179Z I0308 10:12:20.364549       1 template.go:657] router "msg"="Shutdown requested, waiting 45s for new connections to cease"
    2021-03-08T10:12:23.152004630Z E0308 10:12:23.151954       1 haproxy.go:442] unexpected error while reading CSV: read unix @->/var/lib/haproxy/run/haproxy.sock.22.tmp: i/o timeout
    2021-03-08T10:12:33.436866028Z E0308 10:12:33.436808       1 limiter.go:165] error reloading router: exit status 1
    2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable
    2021-03-08T10:12:33.436866028Z  - Proxy protocol on, checking http://localhost:80 ...
    2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable
    2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable
    2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable
    2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable
    2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable
    2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: Resource temporarily unavailable
    2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable
    2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable
    2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable
    2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable
    2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable
    2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable
    2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable
    2021-03-08T10:12:33.436866028Z  - Exceeded max wait time (30) in health check - 1 retry attempt(s).
    2021-03-08T10:12:34.529593312Z E0308 10:12:34.529538       1 haproxy.go:442] unexpected error while reading CSV: read unix @->/var/lib/haproxy/run/haproxy.sock.22.tmp: i/o timeout
    2021-03-08T10:12:53.147369034Z E0308 10:12:53.147327       1 haproxy.go:442] unexpected error while reading CSV: read unix @->/var/lib/haproxy/run/haproxy.sock.22.tmp: i/o timeout
    2021-03-08T10:12:58.148021030Z E0308 10:12:58.147971       1 haproxy.go:442] unexpected error while reading CSV: read unix @->/var/lib/haproxy/run/haproxy.sock.22.tmp: i/o timeout
    2021-03-08T10:12:59.459645015Z E0308 10:12:59.459598       1 limiter.go:165] error reloading router: exit status 1
    2021-03-08T10:12:59.459645015Z [WARNING] 066/101258 (103533) : Failed to get the number of sockets to be transferred !
    2021-03-08T10:12:59.459645015Z [ALERT] 066/101258 (103533) : Failed to get the sockets from the old process!
    2021-03-08T10:13:04.525933239Z E0308 10:13:04.525878       1 haproxy.go:442] unexpected error while reading CSV: read unix @->/var/lib/haproxy/run/haproxy.sock.22.tmp: i/o timeout
    2021-03-08T10:13:05.364743081Z I0308 10:13:05.364698       1 template.go:659] router "msg"="Instructing the template router to terminate"
    2021-03-08T10:13:23.147468863Z E0308 10:13:23.147415       1 haproxy.go:442] unexpected error while reading CSV: read unix @->/var/lib/haproxy/run/haproxy.sock.22.tmp: i/o timeout
    2021-03-08T10:13:34.525511669Z E0308 10:13:34.525459       1 haproxy.go:442] unexpected error while reading CSV: read unix @->/var/lib/haproxy/run/haproxy.sock.22.tmp: i/o timeout
    2021-03-08T10:13:35.516066046Z E0308 10:13:35.516011       1 template.go:661] router "msg"="Router did not shut down cleanly" "error"="error reloading router: exit status 1\n - Shutting down\nerror: Some processes did not exit within 30s\n"
    

Environment

  • Red Hat OpenShift Container Platform 4.6

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content