Idle time increases if number of running bash processes exceeds number of cores.

Issue

• high CPU utilization during process

Using the following bash script:

# cat bashtest.sh 
until [ 1 = 2 ]; do let x=65535*65535*65535*65535*65535*65535*65535; done

On a system with N cores running N instances of the above script the idle time drops to zero. Running one more copy causes the idle time to increase.

An idle system with 2 cores.

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 595472  16444 173728    0    0    58     7  507  870 73  8 19  1  0
 0  0      0 595472  16444 173728    0    0     0    28 1027   35  0  0 100  0  0
 0  0      0 595472  16444 173728    0    0     0     0 1005   30  0  0 100  0  0
 0  0      0 595472  16444 173728    0    0     0     0 1022   29  0  0 100  0  0
 0  0      0 595472  16444 173728    0    0     0     0 1003   24  0  0 100  0  0
...

Two copies of the script started:

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
...
 0  1      0 595472  16448 173724    0    0     0    20 1022   32  0  0 100  1  0
 0  0      0 595472  16452 173728    0    0     0     4 1007   34  0  0 98  3  0
 1  0      0 595340  16452 173728    0    0     0     0 1029   44 14  2 84  0  0
 1  0      0 595340  16452 173728    0    0     0     0 1012   38 45  5 50  0  0
 1  0      0 595340  16452 173728    0    0     0     0 1029   50 45  5 50  0  0
 2  0      0 595332  16452 173728    0    0     0     0 1013 1774 67  8 25  0  0
 2  0      0 595332  16452 173728    0    0     0     0 1020 2725 87 11  2  0  0
 2  0      0 595332  16460 173720    0    0     0    56 1004 3267 87 10  2  0  0
...

Idle time is reduced as the system is kept busy. Two more copies started:

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
...
 3  0      0 595332  16476 173720    0    0     0    12 1002 3061 85 10  4  0  0
 2  0      0 595332  16476 173720    0    0     0     0 1018 2888 85 11  4  0  0
 3  0      0 594952  16476 173728    0    0     0     0 1015 1216 62  7 31  0  0
 4  0      0 594952  16476 173728    0    0     0     0 1019 1667 67  8 26  0  0
 3  0      0 594696  16476 173728    0    0     0     0 1011 1648 66  7 27  0  0
 2  0      0 594696  16476 173728    0    0     0     0 1020 1336 59  7 34  0  0
 4  0      0 594696  16484 173720    0    0     0    56 1006 1858 71  7 22  0  0
...

Idle time increases sharply.