Server is slow and kswapd is hogging cpu due to pre-existing load over CPU

Issue

• Following outputs show Kswapd0 daemon consuming a lot of cpu alongwith other java daemons.

• System is using PAE kernel:

$ cat uname.out 
Linux svbgcs 2.6.18-238.el5PAE #1 SMP Sun Dec 19 14:42:44 EST 2010 i686 i686 i386 GNU/Linux

• System is not facing much load average. 4 CPUs are available in dmesg, so Load average below 3 is acceptable.

$ less top.out | grep "load average"
top - 14:43:05 up 14:22, 10 users,  load average: 2.09, 1.67, 1.55
top - 14:43:08 up 14:22, 10 users,  load average: 2.09, 1.67, 1.55
top - 14:43:11 up 14:22, 10 users,  load average: 2.56, 1.78, 1.59
top - 14:43:14 up 14:22, 10 users,  load average: 2.60, 1.80, 1.60
top - 14:43:17 up 14:22, 10 users,  load average: 2.60, 1.80, 1.60

• As seen above system is not having much load, but certainly is facing some high iowait and %sy utilization

$ less top.out | grep "Cpu(s)"
Cpu(s):  1.0%us, 13.5%sy,  0.0%ni, 68.3%id, 16.9%wa,  0.0%hi,  0.1%si,  0.0%st
Cpu(s):  0.5%us,  4.1%sy,  0.0%ni, 70.1%id, 25.2%wa,  0.0%hi,  0.1%si,  0.0%st
Cpu(s):  0.3%us, 20.4%sy,  0.0%ni, 48.9%id, 30.2%wa,  0.1%hi,  0.1%si,  0.0%st
Cpu(s):  0.4%us, 18.8%sy,  0.0%ni, 50.6%id, 30.1%wa,  0.1%hi,  0.0%si,  0.0%st
Cpu(s):  0.2%us,  9.0%sy,  0.0%ni, 67.7%id, 23.0%wa,  0.0%hi,  0.1%si,  0.0%st

• The %sy utilization seems to be due to below kswapd0 daemon, which is going in D state frequently, and the Java process probably facing high iowait.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  ------------------------------------------------------------------
  258 root      35  19     0    0    0 D 47.2  0.0 153:59.65 kswapd0
  258 root      35  19     0    0    0 R 13.0  0.0 154:00.05 kswapd0
  258 root      35  19     0    0    0 D 29.1  0.0 154:00.93 kswapd0
  258 root      35  19     0    0    0 R 33.9  0.0 154:01.96 kswapd0
  258 root      34  19     0    0    0 D 24.2  0.0 154:02.70 kswapd0          

  5991 root      16   0 1286m 8784 1780 S 18.5  0.1  14:20.22 java
  5991 root      16   0 1286m  12m 1780 S 24.7  0.2  14:20.97 java
  5991 root      16   0 1286m  13m 1888 S 12.8  0.2  14:21.36 java


procs -----------memory---------- ---swap-- -----io----   --system-- -----cpu------
 r  b   swpd   free     buff  cache   si   so    bi    bo   in   cs us sy id wa st
-----------------------------------------------------------------------------------
14  1 1042288 314088    592 7755576 7756    0  8044   636 1726 2508  0 38 41 21  0
 0  4 1039912 314336    952 7757156 5256    0  7156   180 1590 2190  0  6 68 26  0
 2  1 1039912 313888    932 7757752 4776    0  5060  1108 1685 4582  1 22 49 28  0
 1  2 1039912 314704    808 7756124 4956    0  5084   792 1467 1840  0 10 66 24  0
 2  0 1039912 314956    712 7755648 3360    0  3384   756 1439 1743  1 29 45 25  0
 0  2 1039912 315136    568 7755452 1756    0  1800     8 1119 1357  0  4 67 28  0
 8  1 1039912 314236    504 7755372 3856    0  3920  1240 1462 1899  0 60 14 26  0
18  1 1039912 314312    428 7755404 3144    0  3160  1168 1389 1776  0 64 20 16  0

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache     si   so    bi    bo   in   cs us sy id wa st
-----------------------------------------------------------------------------------
 1  0 1040200 328228    672 7754608  294  126   755   224   62  161  1 14 68 17  0
 2  0 1040200 329824    568 7752428  744    0  1072   408 1106 1376  0 15 73 12  0
 1  0 1040200 327260    560 7754256  860    0  2056     0 1223 1692  0  5 68 26  0
 0  3 1040420 333788    536 7754264  104 3928   560  3988 1681 1416  0  8 75 18  0
 0  0 1040440 331588    548 7755972  764  192  1788   192 1306 1673  0  0 90 10  0
 0  1 1040440 330744    584 7756688  124    0   884    40 1041 1256  0  0 96  4  0
 0  1 1040440 330620    592 7756688   56    0    68     4 1029 1242  0  0 99  1  0
 0  0 1040440 330612    604 7756620   56    0    60  1144 1013 1191  0  0 99  1  0
 0  1 1040440 326216    672 7758880 2516    0  4356    40 1410 1900  1  0 87 12  0
 0  0 1040440 324936    684 7759392  408    0   968     0 1177 1545  0  0 97  2  0
 0  3 1040896 326776    680 7760132  532 3068  1280  3072 1468 1321  0  1 86 13  0
 0  3 1041176 334032    656 7758820  156 5168   188  5176 1842 1368  0  6 75 19  0
 0  8 1041176 323916    712 7762120 6600    0 10020    12 1715 2160  1  1 72 26  0
 0  8 1041456 331624    720 7762192  992 7736  1172  7804 1868 25924  0  2 74 24  0
 0  4 1042112 328508    776 7764876 3432 2464  5984  2480 1766 5178  1  1 72 26  0
 1  5 1042552 328360    788 7767384 2664 4512  5028  4516 1650 1718  0  2 72 26  0
 0  6 1042556 333744    788 7767764  524 4944   780  4944 1888 1313  0  2 75 24  0
 0  2 1042408 327268   1084 7770944 3100    0  6396   104 1513 2039  0  0 75 25  0
 0  3 1042544 324868   1328 7771184 3612 2296  4344  2296 1487 4646  1  1 74 24  0
 2  4 1043160 332116   1340 7771472  296 6100   320  6372 1791 1292  0  1 65 34  0
 0  2 1043160 329188   1408 7772600 3052  708  4208   872 1502 1551  0  1 72 28  0