Looking to see if anyone else is facing the same problem.

I have a Tower cluster of 3 nodes, and an external database in a firewalled off segment. When Tower starts, everything is fine. When the firewall timeout on a connection is applied (30 minutes), I get job output capture/display failures with a ton of "connection already closed" errors in the callback_receiver.log. Which makes sense: the database connection that Tower is expecting to be there is torn down.

What doesn't make so much sense is: this seems like aberrant behavior on behalf of .. django or psycopg. If the database gets rebooted (or if the cluster fails over), it looks like Tower will never figure that out and it'll take a Tower restart to restore functionality.

I'm not sure I can get the firewall out from in front of the database ("security"), so is there a mechanism available somewhere in the software stack to allow the connections to be tested and then rebuilt if it's down?

--Chris