Datasource takes too long to recover when a DB failover occurs

Solution Unverified - Updated -

Issue

When a DB server suddenly goes down and fails over without sending RST packet (ex. AWS RDS), the JDBC connections pooled in the datasource remain in a corrupted state but are not evicted from the pool.
These connections will fail with a timeout error when the application attempts to use them next time.

To prevent this, Quarkus datasource support connection validation either in the background(quarkus.datasource.jdbc.background-validation-interval) or at the time the connection is pulled from the pool (quarkus.datasource.jdbc.validate-on-borrow).
However, in the current implementation, no query timeout is set for the validation query, causing it to wait until the standard socket timeout is reached.

 Apr 01 11:02:09 rhbk kc.sh[1786]: 2025-04-01 11:02:09,862 WARN  [io.quarkus.agroal.runtime.DataSources] (org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker) Connection validation failed: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
Apr 01 11:02:09 rhbk kc.sh[1786]:         at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:399)
Apr 01 11:02:09 rhbk kc.sh[1786]:         at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:517)
Apr 01 11:02:09 rhbk kc.sh[1786]:         at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:434)

[omitted]

Apr 01 11:02:09 rhbk kc.sh[1786]: Caused by: java.net.SocketTimeoutException: Read timed out
Apr 01 11:02:09 rhbk kc.sh[1786]:         at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:288)
Apr 01 11:02:09 rhbk kc.sh[1786]:         at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:314)
Apr 01 11:02:09 rhbk kc.sh[1786]:         at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355)
Apr 01 11:02:09 rhbk kc.sh[1786]:         at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808)
Apr 01 11:02:09 rhbk kc.sh[1786]:         at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
Apr 01 11:02:09 rhbk kc.sh[1786]:         at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:192)
Apr 01 11:02:09 rhbk kc.sh[1786]:         at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:159)
Apr 01 11:02:09 rhbk kc.sh[1786]:         at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:144)
Apr 01 11:02:09 rhbk kc.sh[1786]:         at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:76)
Apr 01 11:02:09 rhbk kc.sh[1786]:         at org.postgresql.core.PGStream.receiveChar(PGStream.java:476)
Apr 01 11:02:09 rhbk kc.sh[1786]:         at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2174)
Apr 01 11:02:09 rhbk kc.sh[1786]:         at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:372)
Apr 01 11:02:09 rhbk kc.sh[1786]:         ... 52 more

Environment

  • Red Hat Build of Keycloak (RHBK)
    • 22.0
    • 24.0
    • 26.0
  • Red Hat Build of Quarkus (RHBQ)
    • 3.15

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content