Connections to Oracle DB server gets timed out with 'EAGAIN (Resource temporarily unavailable)' errors

Issue

• We have 12 application servers creating 500 TCP connections to the single Oracle RAC DB server (total ~6000 connections). The TCP connection to DB server is failing periodically, the strace logs shows that TCP connection to server is failing with below errors:

  $ strace -p 32120 |grep -i "Resource temporarily unavailable"
  [...]
  474   <... futex resumed> )             = -1 EAGAIN (Resource temporarily unavailable)
  932   <... futex resumed> )             = -1 EAGAIN (Resource temporarily unavailable)
  32461 <... futex resumed> )             = -1 EAGAIN (Resource temporarily unavailable)
  32462 <... futex resumed> )             = -1 EAGAIN (Resource temporarily unavailable)
  32466 <... futex resumed> )             = -1 EAGAIN (Resource temporarily unavailable)
  32466 <... futex resumed> )             = -1 EAGAIN (Resource temporarily unavailable)
  32461 <... futex resumed> )             = -1 EAGAIN (Resource temporarily unavailable)
  

• There were no messages observed on console or in /var/log/messages file when TCP connections were timed out.