Issue

• On HP DL580 server with 80 cpu threads, the average load is

56.9%us,  8.1%sy,  0.0%ni, 33.6%id,  0.0%wa,  0.1%hi,  1.3%si,  0.0%st

• The load is stable but suddenly, the system consumes 54% of sys CPU.

42.2%us, 54.0%sy,  0.0%ni,  2.9%id,  0.0%wa,  0.0%hi,  0.9%si,  0.0%st

although the load hasn't changed on the server.

• When using oprofile to analyse the calls during this situation, we see a high semop rate.
  1) Just BEFORE we enter in the problem:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 62.53    0.026359           3     10017           read
 11.90    0.005017           0     20040           semop
  9.69    0.004083           0     20034           futex
  6.25    0.002635           0     10017           write
  4.52    0.001907           1      2690           sendto
  3.16    0.001330           0      5838           recvfrom
  1.78    0.000750           0      6760           clock_gettime
  0.17    0.000071           0       152           poll
  0.00    0.000000           0         3         3 open
  0.00    0.000000           0         1         1 stat
------ ----------- ----------- --------- --------- ----------------
100.00    0.042152                 75552         4 total

2) During the problem :

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 98.87    2.014801         149     13545           semop
  0.70    0.014197           2      6770           read
  0.15    0.003097           2      1953           sendto
  0.09    0.001858           0     13539           futex
  0.09    0.001830           0      5426           recvfrom
  0.06    0.001247           0      6770           write
  0.03    0.000583           0      5079           clock_gettime
  0.01    0.000140           1       136           poll
  0.00    0.000000           0         3         3 open
  0.00    0.000000           0         1         1 stat
------ ----------- ----------- --------- --------- ----------------
100.00    2.037753                 53222         4 total

• When we run the exact same program on different HW and RHEL 5.6 we do not see this issue : "on DL380 G6 24vCPUS, we can do ~ 8000 caps (~300 000 semop/s) but on DL580G7 with exact SAME Software, if I do 8000 caps on also 25 seps, I fall down into the semop contention…