OpenShift Router fails with "reload-haproxy: fork: retry: Resource temporarily unavailable" error
Issue
-
On OpenShift Container Platform 4.6, we are seeing the OpenShift Routers crashing with the following error message:
[..] 2021-03-08T10:11:25.812338170Z I0308 10:11:25.812241 1 router.go:569] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" 2021-03-08T10:11:32.308512149Z I0308 10:11:32.308469 1 router.go:569] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" 2021-03-08T10:11:47.451814165Z I0308 10:11:47.451761 1 router.go:569] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" 2021-03-08T10:11:56.483946519Z I0308 10:11:56.483892 1 router.go:569] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n/var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable\n/var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable\n/var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable\n - Health check ok : 0 retry attempt(s).\n" 2021-03-08T10:12:20.364596179Z I0308 10:12:20.364549 1 template.go:657] router "msg"="Shutdown requested, waiting 45s for new connections to cease" 2021-03-08T10:12:23.152004630Z E0308 10:12:23.151954 1 haproxy.go:442] unexpected error while reading CSV: read unix @->/var/lib/haproxy/run/haproxy.sock.22.tmp: i/o timeout 2021-03-08T10:12:33.436866028Z E0308 10:12:33.436808 1 limiter.go:165] error reloading router: exit status 1 2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable 2021-03-08T10:12:33.436866028Z - Proxy protocol on, checking http://localhost:80 ... 2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable 2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable 2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable 2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable 2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable 2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: Resource temporarily unavailable 2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable 2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable 2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable 2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable 2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable 2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable 2021-03-08T10:12:33.436866028Z /var/lib/haproxy/reload-haproxy: fork: retry: Resource temporarily unavailable 2021-03-08T10:12:33.436866028Z - Exceeded max wait time (30) in health check - 1 retry attempt(s). 2021-03-08T10:12:34.529593312Z E0308 10:12:34.529538 1 haproxy.go:442] unexpected error while reading CSV: read unix @->/var/lib/haproxy/run/haproxy.sock.22.tmp: i/o timeout 2021-03-08T10:12:53.147369034Z E0308 10:12:53.147327 1 haproxy.go:442] unexpected error while reading CSV: read unix @->/var/lib/haproxy/run/haproxy.sock.22.tmp: i/o timeout 2021-03-08T10:12:58.148021030Z E0308 10:12:58.147971 1 haproxy.go:442] unexpected error while reading CSV: read unix @->/var/lib/haproxy/run/haproxy.sock.22.tmp: i/o timeout 2021-03-08T10:12:59.459645015Z E0308 10:12:59.459598 1 limiter.go:165] error reloading router: exit status 1 2021-03-08T10:12:59.459645015Z [WARNING] 066/101258 (103533) : Failed to get the number of sockets to be transferred ! 2021-03-08T10:12:59.459645015Z [ALERT] 066/101258 (103533) : Failed to get the sockets from the old process! 2021-03-08T10:13:04.525933239Z E0308 10:13:04.525878 1 haproxy.go:442] unexpected error while reading CSV: read unix @->/var/lib/haproxy/run/haproxy.sock.22.tmp: i/o timeout 2021-03-08T10:13:05.364743081Z I0308 10:13:05.364698 1 template.go:659] router "msg"="Instructing the template router to terminate" 2021-03-08T10:13:23.147468863Z E0308 10:13:23.147415 1 haproxy.go:442] unexpected error while reading CSV: read unix @->/var/lib/haproxy/run/haproxy.sock.22.tmp: i/o timeout 2021-03-08T10:13:34.525511669Z E0308 10:13:34.525459 1 haproxy.go:442] unexpected error while reading CSV: read unix @->/var/lib/haproxy/run/haproxy.sock.22.tmp: i/o timeout 2021-03-08T10:13:35.516066046Z E0308 10:13:35.516011 1 template.go:661] router "msg"="Router did not shut down cleanly" "error"="error reloading router: exit status 1\n - Shutting down\nerror: Some processes did not exit within 30s\n"
Environment
- Red Hat OpenShift Container Platform 4.6
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.