Suddenly high consumption of sys CPU on DL580 G7 with 80 cpu threads on 4 sockets
Issue
- On HP DL580 server with 80 cpu threads, the average load is
56.9%us, 8.1%sy, 0.0%ni, 33.6%id, 0.0%wa, 0.1%hi, 1.3%si, 0.0%st
- The load is stable but suddenly, the system consumes
54%
of sys CPU.
42.2%us, 54.0%sy, 0.0%ni, 2.9%id, 0.0%wa, 0.0%hi, 0.9%si, 0.0%st
although the load hasn't changed on the server.
- When using oprofile to analyse the calls during this situation, we see a high semop rate.
1) Just BEFORE we enter in the problem:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
62.53 0.026359 3 10017 read
11.90 0.005017 0 20040 semop
9.69 0.004083 0 20034 futex
6.25 0.002635 0 10017 write
4.52 0.001907 1 2690 sendto
3.16 0.001330 0 5838 recvfrom
1.78 0.000750 0 6760 clock_gettime
0.17 0.000071 0 152 poll
0.00 0.000000 0 3 3 open
0.00 0.000000 0 1 1 stat
------ ----------- ----------- --------- --------- ----------------
100.00 0.042152 75552 4 total
2) During the problem :
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
98.87 2.014801 149 13545 semop
0.70 0.014197 2 6770 read
0.15 0.003097 2 1953 sendto
0.09 0.001858 0 13539 futex
0.09 0.001830 0 5426 recvfrom
0.06 0.001247 0 6770 write
0.03 0.000583 0 5079 clock_gettime
0.01 0.000140 1 136 poll
0.00 0.000000 0 3 3 open
0.00 0.000000 0 1 1 stat
------ ----------- ----------- --------- --------- ----------------
100.00 2.037753 53222 4 total
- When we run the exact same program on different HW and RHEL 5.6 we do not see this issue : "on DL380 G6 24vCPUS, we can do ~ 8000 caps (~300 000 semop/s) but on DL580G7 with exact SAME Software, if I do 8000 caps on also 25 seps, I fall down into the semop contention…
Environment
- Red Hat Enterprise Linux 5.8
- Red Hat Enterprise Linux 6
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.