Suddenly high consumption of sys CPU on DL580 G7 with 80 cpu threads on 4 sockets

Solution Unverified - Updated -

Issue

  • On HP DL580 server with 80 cpu threads, the average load is
56.9%us,  8.1%sy,  0.0%ni, 33.6%id,  0.0%wa,  0.1%hi,  1.3%si,  0.0%st
  • The load is stable but suddenly, the system consumes 54% of sys CPU.
42.2%us, 54.0%sy,  0.0%ni,  2.9%id,  0.0%wa,  0.0%hi,  0.9%si,  0.0%st

although the load hasn't changed on the server.

  • When using oprofile to analyse the calls during this situation, we see a high semop rate.
    1) Just BEFORE we enter in the problem:
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 62.53    0.026359           3     10017           read
 11.90    0.005017           0     20040           semop
  9.69    0.004083           0     20034           futex
  6.25    0.002635           0     10017           write
  4.52    0.001907           1      2690           sendto
  3.16    0.001330           0      5838           recvfrom
  1.78    0.000750           0      6760           clock_gettime
  0.17    0.000071           0       152           poll
  0.00    0.000000           0         3         3 open
  0.00    0.000000           0         1         1 stat
------ ----------- ----------- --------- --------- ----------------
100.00    0.042152                 75552         4 total

2) During the problem :

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 98.87    2.014801         149     13545           semop
  0.70    0.014197           2      6770           read
  0.15    0.003097           2      1953           sendto
  0.09    0.001858           0     13539           futex
  0.09    0.001830           0      5426           recvfrom
  0.06    0.001247           0      6770           write
  0.03    0.000583           0      5079           clock_gettime
  0.01    0.000140           1       136           poll
  0.00    0.000000           0         3         3 open
  0.00    0.000000           0         1         1 stat
------ ----------- ----------- --------- --------- ----------------
100.00    2.037753                 53222         4 total
  • When we run the exact same program on different HW and RHEL 5.6 we do not see this issue : "on DL380 G6 24vCPUS, we can do ~ 8000 caps (~300 000 semop/s) but on DL580G7 with exact SAME Software, if I do 8000 caps on also 25 seps, I fall down into the semop contention…

Environment

  • Red Hat Enterprise Linux 5.8
  • Red Hat Enterprise Linux 6

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content