SCHED_OTHER not properly balancing load when CPU affinity groups specified

Issue

SCHED_OTHER not properly balancing load when CPU affinity groups specified.

One way this has been demonstrated is by using the unsupported stress binary from EPEL to spawn a child process for each CPU in the affinity group. But as can see from the following output, one of the CPUs is left idle (CPU 30 here).

# CPUs=29,30,32,34,36,38,39,41
# taskset -c ${CPUs} stress -c $(echo ${CPUs} | awk -F, '{print NF}') &
# LANG=C sar -P ${CPUs}  1 | awk '$1!="" && $2!="CPU" && $3<99 {print $0}'
Linux 3.10.0-327.el7 (hostname.redhat.com)  12/29/15    _x86_64_    (60 CPU)
13:24:28         30      0.00      0.00      0.00      0.00      0.00    100.00
13:24:29         30      0.00      0.00      0.00      0.00      0.00    100.00
13:24:30         30      0.00      0.00      0.00      0.00      0.00    100.00
13:24:31         30      0.00      0.00      0.00      0.00      0.00    100.00

# ps -C stress -o psr,c,pid,ppid,args | sort -n
PSR  C   PID  PPID COMMAND
 29 50  3850  3845 stress -c 8 <--- 1st child on CPU 29
 29 50  3851  3845 stress -c 8 <--- 2nd child on CPU 29
 32  0  3845  3810 stress -c 8 <--- parent PID, no real cpu usage expected
 32 99  3856  3845 stress -c 8
 34 99  3853  3845 stress -c 8
 36 99  3855  3845 stress -c 8
 38 99  3854  3845 stress -c 8
 39 99  3852  3845 stress -c 8
 41 99  3849  3845 stress -c 8