JBOSS Application Server Heap Memory issues

Latest response

Our Application is getting slower after some period of time in customer sites which consists of Huge DB.

Please find our observations and analysis for the slowness issue observed in Customer sites.

We are facing the performance degrade of the application after 1 Week of Packet restart.
Previously we used jre version 1.7.4 , and it seems to be no performance issues with that.
After upgrading to jre version 1.7.80 , there are lot of performance issues observed after two weeks

Previously the JBOSS parameters are
-server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms1024m -Xmx1536m -XX:MaxPermSize=384m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml -Xss256k -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:-UseSplitVerifier -XX:+HeapDumpOnOutOfMemoryError -XX:ReservedCodeCacheSize=128m -Dorg.jboss.server.bootstrap.maxThreads=8

1) We removed the JBOSS parameter "-XX:+TieredCompilation" and restarted the application.
It works fine without any performance degradation for 15 days
After removing the Tiered Compilation also, the application becoming slow after 15 days of restart.
We have seen the Old Gen Space is becoming full and the Garbage Collector Threads are running continuously.
concurrent mark-sweep generation:
capacity = 1073741824 (1024.0MB)
used = 1073741824 (1024.0MB)
free = 0 (0.0MB)
100.0% used

2) Later we added following JBOSS tuning parameters
"-XX:+UseCompressedOops","-XX:CMSInitiatingOccupancyFraction=70","-XX:+UseCMSInitiatingOccupancyOnly","-XX:NewRatio=3"
and increased the heap size as "-Xms1536m -Xmx2048m"

   Present the JBOSS Parameters are as follows. 

       -server -XX:+UseCompressedOops -Xms1536m -Xmx2048m -XX:MaxPermSize=384m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000                         -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml -Xss256k -XX:+UseConcMarkSweepGC         -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:-UseSplitVerifier 

   After 4 days , concurrent mark-sweep generation becoming 80% the Garbage Collector Threads are running continuously and the CPU Utilization for iptnms PID for the application is 600-700% all the time. 

    concurrent mark-sweep generation: 
    capacity = 1610612736 (1536.0MB) 
    used     = 1348910168 (1286.4209823608398MB) 
    free     = 261702568 (249.57901763916016MB) 
    83.75136603911717% used

3)After adding above parameters , application getting slower again after 2 days.
concurrent mark-sweep generation:
capacity = 1610612736 (1536.0MB)
used = 1348910168 (1286.4209823608398MB)
free = 261702568 (249.57901763916016MB)
83.75136603911717% used

Now we removed these parameters " -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 ".
we thought that once concurrent-mark-sweep generation crosses 70% , garbage collector trying to clear objects in Heap memory parallely. So we removed those parameters.

Present the JBOSS Parameters are as follows.
-server -XX:+UseCompressedOops -Xms1536m -Xmx2048m -XX:MaxPermSize=384m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml -Xss256k -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -XX:-UseSplitVerifier .

Heap memory Information file is attached.
concurrent-mark-sweep generation crosses 70% and after that 1 day no users worked on application and again it automatically comes down to 56%. Some of these findings can be observed in attached file.

Please post your suggestions/solutions for this issue as soon as possible. Customers are facing problems with application slowness.

Attachments

AT

Responses