RHEL5 cpusets when used with isolcpus doesn't distribute threads across groups what's the best way to prevent a set of cpus from getting interrupted

Issue

I'm trying to make sure the cpusets I'm creating with csets don't get interrupted by other processes. In this example, I'm creating a cpuset with all the cores in one node and then assigning the threads of router to it:

# cset set --cpu=8-15,40-47 --mem=0-3 ROUTER
# for i in `ps -ef | grep router`; do sudo cset proc --move $i --threads ROUTER; done

This seems to work fine (as you can see, some router threads spin at 100% cpu)

[root@acme123 ~]#  ps -eLo pcpu,psr,pid,user,args  | grep router
arcaqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
99.6  44  2486 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
99.6  45  2486 acmeqa  /app/utp/utp/bin/ux_router SOD -name EQROUTER
99.6  11  2486 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
99.6  10  2486 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
99.6  14  2486 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
99.6  40  2486 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
 1.5   8  2486 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
 0.0  41  2486 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
 0.0   8  2486 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
 0.0   8  2486 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
 0.0   8  2486 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER

However, if I add "isolcpus=8-15,40-47" to the boot line and try the same test, all the router threads get pinned to the first core in the cpuset, causing terrible performance.

1.5   8 27465 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
62.0   8 27465 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
62.0   8 27465 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
62.1   8 27465 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
62.0   8 27465 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
62.0   8 27465 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
62.0   8 27465 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
 1.9   8 27465 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
 0.0   8 27465 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
 0.0   8 27465 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
 0.0   8 27465 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER
 0.0   8 27465 acmeqa   /app/utp/utp/bin/ux_router SOD -name EQROUTER