Data Science Pipelines unable to access mariadb instance within same namespace

Issue

We are seeing that we are continuously retrying reach out from the ds pipeline
2024-04-19T17:34:55.646673477Z 2024-04-19T17:34:55Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "xxxxxxx", "dspa_name": "pipelines-definition"}

mariadb database pod is never becoming ready, because:
2024-04-19T16:08:34.141934146Z 2024-04-19 16:08:34 10 [Warning] Aborted connection 10 to db: 'unconnected' user: 'mlpipeline' host: '10.36.10.15' (Got an error reading communication packets)

So it just gets stuck in this "waiting for database to become ready" state.