A mod_cluster worker has -1 load factor and generates 503s when started following an ungraceful shutdown or crash
Issue
- After we have a backend tomcat/JBoss node crash or be ungracefully shutdown, we restart a server under a new VM, which gets a new ip address. We then get persistent 503s from this new node and the new node persists at a -1 load state on the
mod_cluster-manager
page. - We checked the STATUS requests and JBoss is sending positive load through MCMP and the STATUS requests succeed with a 200. We note that pings are failing because they are sent to the ip of the old crashed/ungracefully shutdown worker:
[Thu Aug 08 03:03:33 2013] [debug] mod_manager.c(1442): manager_trans STATUS (/)
[Thu Aug 08 03:03:33 2013] [debug] mod_manager.c(1957): manager_handler STATUS (/) processing: "JVMRoute=node1&Load=91"
[Thu Aug 08 03:03:33 2013] [debug] proxy_util.c(2011): proxy: http: has acquired connection for (oldworkerip)
[Thu Aug 08 03:03:33 2013] [debug] proxy_util.c(2067): proxy: connecting http://oldworkerip:8080/ to oldworkerip:80
[Thu Aug 08 03:03:33 2013] [debug] proxy_util.c(2193): proxy: connected http://oldworkerip:8080/ to oldworkerip:8080
[Thu Aug 08 03:03:33 2013] [debug] proxy_util.c(2444): proxy: http: fam 2 socket created to connect to oldworkerip
[Thu Aug 08 03:03:33 2013] [error] (111)Connection refused: proxy: http: attempt to connect to oldworkerip:8080 (oldworkerip:8080) failed
[Thu Aug 08 03:03:33 2013] [error] ap_proxy_connect_backend disabling worker for oldworkerip
[Thu Aug 08 03:03:33 2013] [debug] mod_proxy_cluster.c(1422): proxy_cluster_try_pingpong: can't connect to backend
[Thu Aug 08 03:03:33 2013] [debug] proxy_util.c(2029): proxy: http: has released connection for (oldworkerip)
[Thu Aug 08 03:03:33 2013] [debug] mod_proxy_cluster.c(1496): proxy_cluster_isup: pingpong failed
[Thu Aug 08 03:03:33 2013] [debug] mod_manager.c(1998): manager_handler STATUS OK
Environment
- JBoss Enterprise Application Platform 5
- JBoss Enterprise Web Server (EWS) 1.x
- mod_cluster 1.0.10.GA_CP04
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.