Datasource takes too long to recover when a DB failover occurs
Issue
When a DB server suddenly goes down and fails over without sending RST packet (ex. AWS RDS), the JDBC connections pooled in the datasource remain in a corrupted state but are not evicted from the pool.
These connections will fail with a timeout error when the application attempts to use them next time.
To prevent this, Quarkus datasource support connection validation either in the background(quarkus.datasource.jdbc.background-validation-interval) or at the time the connection is pulled from the pool (quarkus.datasource.jdbc.validate-on-borrow).
However, in the current implementation, no query timeout is set for the validation query, causing it to wait until the standard socket timeout is reached.
Apr 01 11:02:09 rhbk kc.sh[1786]: 2025-04-01 11:02:09,862 WARN [io.quarkus.agroal.runtime.DataSources] (org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker) Connection validation failed: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
Apr 01 11:02:09 rhbk kc.sh[1786]: at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:399)
Apr 01 11:02:09 rhbk kc.sh[1786]: at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:517)
Apr 01 11:02:09 rhbk kc.sh[1786]: at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:434)
[omitted]
Apr 01 11:02:09 rhbk kc.sh[1786]: Caused by: java.net.SocketTimeoutException: Read timed out
Apr 01 11:02:09 rhbk kc.sh[1786]: at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:288)
Apr 01 11:02:09 rhbk kc.sh[1786]: at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:314)
Apr 01 11:02:09 rhbk kc.sh[1786]: at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355)
Apr 01 11:02:09 rhbk kc.sh[1786]: at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808)
Apr 01 11:02:09 rhbk kc.sh[1786]: at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
Apr 01 11:02:09 rhbk kc.sh[1786]: at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:192)
Apr 01 11:02:09 rhbk kc.sh[1786]: at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:159)
Apr 01 11:02:09 rhbk kc.sh[1786]: at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:144)
Apr 01 11:02:09 rhbk kc.sh[1786]: at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:76)
Apr 01 11:02:09 rhbk kc.sh[1786]: at org.postgresql.core.PGStream.receiveChar(PGStream.java:476)
Apr 01 11:02:09 rhbk kc.sh[1786]: at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2174)
Apr 01 11:02:09 rhbk kc.sh[1786]: at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:372)
Apr 01 11:02:09 rhbk kc.sh[1786]: ... 52 more
Environment
- Red Hat Build of Keycloak (RHBK)
- 22.0
- 24.0
- 26.0
- Red Hat Build of Quarkus (RHBQ)
- 3.15
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.