MPSTAT

Latest response

Hi,

Im running RHEL 5.5 with the shipped Kernel / 16 core system  - cant update kernel because of the application.

when i run mpstat -P ALL , i notice INTR are only one CPU0

 

Shouldnt the INTR be splitted across all CPU? or im missing something here ?

 

01:59:57 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
01:59:58 PM  all    0.06    0.00    1.87    0.19    0.12    1.12    0.00   96.63  40389.90
01:59:58 PM    0    0.00    0.00   14.14    1.01    2.02   16.16    0.00   66.67  40396.97
01:59:58 PM    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
01:59:58 PM    2    0.00    0.00    1.00    0.00    0.00    0.00    0.00   99.00      0.00
01:59:58 PM    3    0.00    0.00    1.00    0.00    0.00    0.00    0.00   99.00      0.00
01:59:58 PM    4    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
01:59:58 PM    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
01:59:58 PM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
01:59:58 PM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
01:59:58 PM    8    0.00    0.00    9.09    1.01    0.00    2.02    0.00   87.88      0.00
01:59:58 PM    9    0.00    0.00    1.00    0.00    0.00    0.00    0.00   99.00      0.00
01:59:58 PM   10    0.00    0.00    0.99    0.00    0.00    0.00    0.00   99.01      0.00
01:59:58 PM   11    0.00    0.00    0.99    0.00    0.00    0.99    0.00   98.02      0.00
01:59:58 PM   12    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
01:59:58 PM   13    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
01:59:58 PM   14    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.00
01:59:58 PM   15    0.00    0.00    0.99    0.00    0.00    0.00    0.00   99.01      0.00

 

Responses

Hello,

You can check the contents of /proc/interrupts to see how your IRQs are distributed amongst the CPUs. 

 

Is it possible that irqbalance is disabled, or that you modified /etc/sysconfig/irqbalance?  Try running:

 

 # chkconfig --list irqbalance

 # ps aux | grep irqbalance

 # rpm -V irqbalance

 

Regards,

John Ruemker, RHCA

Red Hat Technical Account Manager

Online User Groups Moderator

This will probably come of sounding smart-assed, but it's something I've been burned by, more than once, over the years.

 

Whenever I see a post that says "I can't patch/update because of 'application X'", I have to wonder whether the software is some kind of antiquated/legacy software they're attempting to run. Haven't really run into that, before, on Linux, but it was something I frequently had to deal with on Solaris, AIX and HP/UX.

 

 

If you are running an old application, you need to remember that, just because you toss it on new hardware doesn't mean that it will know how to make use of that hardware. All of that begs the question, "is your application multi-CPU aware?" If it's not multi-CPU aware, is there any chance you can run multiple instances of it and then spread your workload across those instances?

I found a decent resource on Linux interrupt dispatching and it is available here:

http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-linux

An interesting statement:

"By default APIC delivers ALL interrupts to core 0."

Apparently having CONFIG_HOTPLUG enabled in your kernel may affect interrupt dispatching as well as not having the APIC enabled in the BIOS of your host.

Hi,

For some reason irqbalance was not turned on, i thought it was, ,my bad then.

Ill turn it on.

Thomas - you are correct from the application side however I think HBA / 10GB NIC should be irq balanced across multi-cpu hardware to better balance the workload.

 

 Tomas: The software we are running is new and should be balanced - good remark though - we had a problem with old software on new hardware.in the past.

 

Thanks

-E