Server is slow and kswapd is hogging cpu due to pre-existing load over CPU
Issue
-
Following outputs show Kswapd0 daemon consuming a lot of cpu alongwith other java daemons.
-
System is using PAE kernel:
$ cat uname.out
Linux svbgcs 2.6.18-238.el5PAE #1 SMP Sun Dec 19 14:42:44 EST 2010 i686 i686 i386 GNU/Linux
- System is not facing much load average. 4 CPUs are available in dmesg, so Load average below 3 is acceptable.
$ less top.out | grep "load average"
top - 14:43:05 up 14:22, 10 users, load average: 2.09, 1.67, 1.55
top - 14:43:08 up 14:22, 10 users, load average: 2.09, 1.67, 1.55
top - 14:43:11 up 14:22, 10 users, load average: 2.56, 1.78, 1.59
top - 14:43:14 up 14:22, 10 users, load average: 2.60, 1.80, 1.60
top - 14:43:17 up 14:22, 10 users, load average: 2.60, 1.80, 1.60
- As seen above system is not having much load, but certainly is facing some high iowait and %sy utilization
$ less top.out | grep "Cpu(s)"
Cpu(s): 1.0%us, 13.5%sy, 0.0%ni, 68.3%id, 16.9%wa, 0.0%hi, 0.1%si, 0.0%st
Cpu(s): 0.5%us, 4.1%sy, 0.0%ni, 70.1%id, 25.2%wa, 0.0%hi, 0.1%si, 0.0%st
Cpu(s): 0.3%us, 20.4%sy, 0.0%ni, 48.9%id, 30.2%wa, 0.1%hi, 0.1%si, 0.0%st
Cpu(s): 0.4%us, 18.8%sy, 0.0%ni, 50.6%id, 30.1%wa, 0.1%hi, 0.0%si, 0.0%st
Cpu(s): 0.2%us, 9.0%sy, 0.0%ni, 67.7%id, 23.0%wa, 0.0%hi, 0.1%si, 0.0%st
- The %sy utilization seems to be due to below kswapd0 daemon, which is going in D state frequently, and the Java process probably facing high iowait.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
------------------------------------------------------------------
258 root 35 19 0 0 0 D 47.2 0.0 153:59.65 kswapd0
258 root 35 19 0 0 0 R 13.0 0.0 154:00.05 kswapd0
258 root 35 19 0 0 0 D 29.1 0.0 154:00.93 kswapd0
258 root 35 19 0 0 0 R 33.9 0.0 154:01.96 kswapd0
258 root 34 19 0 0 0 D 24.2 0.0 154:02.70 kswapd0
5991 root 16 0 1286m 8784 1780 S 18.5 0.1 14:20.22 java
5991 root 16 0 1286m 12m 1780 S 24.7 0.2 14:20.97 java
5991 root 16 0 1286m 13m 1888 S 12.8 0.2 14:21.36 java
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
-----------------------------------------------------------------------------------
14 1 1042288 314088 592 7755576 7756 0 8044 636 1726 2508 0 38 41 21 0
0 4 1039912 314336 952 7757156 5256 0 7156 180 1590 2190 0 6 68 26 0
2 1 1039912 313888 932 7757752 4776 0 5060 1108 1685 4582 1 22 49 28 0
1 2 1039912 314704 808 7756124 4956 0 5084 792 1467 1840 0 10 66 24 0
2 0 1039912 314956 712 7755648 3360 0 3384 756 1439 1743 1 29 45 25 0
0 2 1039912 315136 568 7755452 1756 0 1800 8 1119 1357 0 4 67 28 0
8 1 1039912 314236 504 7755372 3856 0 3920 1240 1462 1899 0 60 14 26 0
18 1 1039912 314312 428 7755404 3144 0 3160 1168 1389 1776 0 64 20 16 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
-----------------------------------------------------------------------------------
1 0 1040200 328228 672 7754608 294 126 755 224 62 161 1 14 68 17 0
2 0 1040200 329824 568 7752428 744 0 1072 408 1106 1376 0 15 73 12 0
1 0 1040200 327260 560 7754256 860 0 2056 0 1223 1692 0 5 68 26 0
0 3 1040420 333788 536 7754264 104 3928 560 3988 1681 1416 0 8 75 18 0
0 0 1040440 331588 548 7755972 764 192 1788 192 1306 1673 0 0 90 10 0
0 1 1040440 330744 584 7756688 124 0 884 40 1041 1256 0 0 96 4 0
0 1 1040440 330620 592 7756688 56 0 68 4 1029 1242 0 0 99 1 0
0 0 1040440 330612 604 7756620 56 0 60 1144 1013 1191 0 0 99 1 0
0 1 1040440 326216 672 7758880 2516 0 4356 40 1410 1900 1 0 87 12 0
0 0 1040440 324936 684 7759392 408 0 968 0 1177 1545 0 0 97 2 0
0 3 1040896 326776 680 7760132 532 3068 1280 3072 1468 1321 0 1 86 13 0
0 3 1041176 334032 656 7758820 156 5168 188 5176 1842 1368 0 6 75 19 0
0 8 1041176 323916 712 7762120 6600 0 10020 12 1715 2160 1 1 72 26 0
0 8 1041456 331624 720 7762192 992 7736 1172 7804 1868 25924 0 2 74 24 0
0 4 1042112 328508 776 7764876 3432 2464 5984 2480 1766 5178 1 1 72 26 0
1 5 1042552 328360 788 7767384 2664 4512 5028 4516 1650 1718 0 2 72 26 0
0 6 1042556 333744 788 7767764 524 4944 780 4944 1888 1313 0 2 75 24 0
0 2 1042408 327268 1084 7770944 3100 0 6396 104 1513 2039 0 0 75 25 0
0 3 1042544 324868 1328 7771184 3612 2296 4344 2296 1487 4646 1 1 74 24 0
2 4 1043160 332116 1340 7771472 296 6100 320 6372 1791 1292 0 1 65 34 0
0 2 1043160 329188 1408 7772600 3052 708 4208 872 1502 1551 0 1 72 28 0
Environment
- Red Hat Enterprise Linux 5.6
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
