How to determine whether smp_affinity can improve network throughput and adjust it accordingly on RHEL6

Issue

• Network throughput has maxed out at a lower throughput than expected. At the same time a CPU is running at 90-100% for softirqs while others are idle.

• Getting less throughput after upgrading from RHEL5 to RHEL6.

• I need to modify the value in the value /proc/irq/#IRQ/smp_affinity
  but I don't understand the mask values for a specific cpu or set of cpus.

• The problem is to do with the network IO flattening after a certain point. We see that %softirq will touch the +90% mark and network IO will flatten out. It won't burst on client requests and there will be an io wait on the client side. Now after we switched irqbalance off and tuned the TCP stack - we do not see %softirq increasing. But we don't see the network getting to a level where it was on RHEL5.