java.net.SocketException: Too many open files

Issue

• We are receiving the 'Too many open files' exception in our server logs:

  Socket accept failed java.net.SocketException: Too many open files
          at java.net.PlainSocketImpl.socketAccept(Native Method)
          at java.net.PlainSocketImpl.accept(Unknown Source)
          at java.net.ServerSocket.implAccept(Unknown Source)
          ...
          ...
  java.io.FileNotFoundException: filename (Too many open files)
          at java.io.FileInputStream.open(Native Method)
          at java.io.FileInputStream.<init>(FileInputStream.java:106)
          at java.io.FileInputStream.<init>(FileInputStream.java:66)
          at java.io.FileReader.<init>(FileReader.java:41)
  

• java.net.SocketException: Too many open files -- Need RCA

  <Critical> <Server> <BEA-002616> <Failed to listen on channel "Default[2]" on 127.0.0.1:7001, failure count: 1, failing for 0 seconds, java.net.SocketException: Too many open files>
  

• Facing the below error on server :

  19:57:22,988 ERROR [DiskStore] ticketCache: Could not create disk store. Initial cause was /tmp/path/ticketCache.data (Too many open files)
  java.io.FileNotFoundException: /tmp/path/ticketCache.data (Too many open files)
      at java.io.RandomAccessFile.open(Native Method)
      at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
      at net.sf.ehcache.store.DiskStore.initialiseFiles(DiskStore.java:208)
      at net.sf.ehcache.store.DiskStore.<init>(DiskStore.java:153)
      at net.sf.ehcache.Cache.createDiskStore(Cache.java:526)
      at net.sf.ehcache.Cache.initialise(Cache.java:503)
      at net.sf.ehcache.CacheManager.addCacheNoCheck(CacheManager.java:634)
      at net.sf.ehcache.CacheManager.addCache(CacheManager.java:624)
      at org.springframework.cache.ehcache.EhCacheFactoryBean.afterPropertiesSet(EhCacheFactoryBean.java:259)
  

• WAR/EAR deployment fails:

  "/deployment=deployment.ear:add(runtime-name="deployment.ear", content=[{"path"=>"/path/to/deployment/deployment.ear","archive"=>false}], enabled=true)"
  {
      "outcome" => "failed",
      "failure-description" => {"WFLYCTL0080: Failed services" => {"jboss.deployment.unit.\"deployment.ear\".STRUCTURE" => "org.jboss.msc.service.StartException in service jboss.deployment.unit.\"deployment.ear\".STRUCTURE: WFLYSRV0153: Failed to process phase STRUCTURE of deployment \"deployment.ear\"
      Caused by: org.jboss.as.server.deployment.DeploymentUnitProcessingException: WFLYEE0054: Failed to process children for EAR [\"/path/to/deployment/deployment.ear\"]
      Caused by: java.io.FileNotFoundException: /jboss-eap-7/standalone/tmp/vfs/deployment/deployment917425a0df51e2b7/library.jar-c42571ac700c1222/library.jar (Too many open files)"}},
      "rolled-back" => true