DBConnectionError on controller node

Issue

• We are seeing DBConnectionError on the controller node. nova-compute service is showing down as well but nova services are up and running. due to that VMs are unable to get the console on this to compute node.

• There're many DVR routers in this environment.

• Here it is quick logs from controller nodes for nova-conductor logs:-

[root@overcloud-controller-2 heat-admin]# tailf /var/log/containers/nova/nova-conductor.log
2020-11-17 17:39:59.861 31 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
2020-11-17 17:39:59.861 31 ERROR oslo_db.sqlalchemy.engines     conn.query(q)
2020-11-17 17:39:59.861 31 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 855, in query
2020-11-17 17:39:59.861 31 ERROR oslo_db.sqlalchemy.engines     self._execute_command(COMMAND.COM_QUERY, sql)
2020-11-17 17:39:59.861 31 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1092, in _execute_command
2020-11-17 17:39:59.861 31 ERROR oslo_db.sqlalchemy.engines     self._write_bytes(packet)
2020-11-17 17:39:59.861 31 ERROR oslo_db.sqlalchemy.engines   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1044, in _write_bytes
2020-11-17 17:39:59.861 31 ERROR oslo_db.sqlalchemy.engines     "MySQL server has gone away (%r)" % (e,))
2020-11-17 17:39:59.861 31 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(110, 'Connection timed out'))") [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)
2020-11-17 17:39:59.861 31 ERROR oslo_db.sqlalchemy.engines