Hi Saurav,

First of all, what app is being used with your JBOSS 5 server? This does seem like a memory leak issue.
While I mention 'java' below, I'm personally suspecting a memory leak (maybe a database or something else) based on the above and what we've seen in our environment... but this may or may not be your case...

I've never used JBOSS, however, another of our admins have had to debug memory leaks with java... In any case, check out these Red Hat solutions
* This one is dated, Red Hat Magazine article from 2006
* This Red Hat solution is from 2014, however it seems it is for JBOSS 6,, but hopefully it may trigger some good ideas to diagnose.
* This Red Hat solution is from 2014, and is for JBOSS 5, and says "BaseClassLoader's blackList causing memory leaks in JBoss"
* This Red Hat Solution is from Dec 2013 has to do with virtual file systems on JBOSS 5
* This Red Hat Solution is from Jan 2014 and discusses Java memory leaks

If none of that helps, another research possibility, here's a list of other solutions at access.redhat.com for "jboss 5 memory leak"

Hopefully one of these articles can help, otherwise perhaps consider how to diagnose java/database or other memory leaks. I have a friend at work who has dealt with diagnosing memory leaks, if needed I can ask him.

Hopefully other community members can post more ideas here.