A mod_cluster worker has -1 load factor and generates 503s when started following an ungraceful shutdown or crash

Solution Verified - Updated -

Issue

  • After we have a backend tomcat/JBoss node crash or be ungracefully shutdown, we restart a server under a new VM, which gets a new ip address. We then get persistent 503s from this new node and the new node persists at a -1 load state on the mod_cluster-manager page.
  • We checked the STATUS requests and JBoss is sending positive load through MCMP and the STATUS requests succeed with a 200. We note that pings are failing because they are sent to the ip of the old crashed/ungracefully shutdown worker:
[Thu Aug 08 03:03:33 2013] [debug] mod_manager.c(1442): manager_trans STATUS (/)
[Thu Aug 08 03:03:33 2013] [debug] mod_manager.c(1957): manager_handler STATUS (/) processing: "JVMRoute=node1&Load=91"
[Thu Aug 08 03:03:33 2013] [debug] proxy_util.c(2011): proxy: http: has acquired connection for (oldworkerip)
[Thu Aug 08 03:03:33 2013] [debug] proxy_util.c(2067): proxy: connecting http://oldworkerip:8080/ to oldworkerip:80
[Thu Aug 08 03:03:33 2013] [debug] proxy_util.c(2193): proxy: connected http://oldworkerip:8080/ to oldworkerip:8080
[Thu Aug 08 03:03:33 2013] [debug] proxy_util.c(2444): proxy: http: fam 2 socket created to connect to oldworkerip
[Thu Aug 08 03:03:33 2013] [error] (111)Connection refused: proxy: http: attempt to connect to oldworkerip:8080 (oldworkerip:8080) failed
[Thu Aug 08 03:03:33 2013] [error] ap_proxy_connect_backend disabling worker for oldworkerip
[Thu Aug 08 03:03:33 2013] [debug] mod_proxy_cluster.c(1422): proxy_cluster_try_pingpong: can't connect to backend
[Thu Aug 08 03:03:33 2013] [debug] proxy_util.c(2029): proxy: http: has released connection for (oldworkerip)
[Thu Aug 08 03:03:33 2013] [debug] mod_proxy_cluster.c(1496): proxy_cluster_isup: pingpong failed
[Thu Aug 08 03:03:33 2013] [debug] mod_manager.c(1998): manager_handler STATUS  OK

Environment

  • JBoss Enterprise Application Platform 5
  • JBoss Enterprise Web Server (EWS) 1.x
  • mod_cluster 1.0.10.GA_CP04

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content