Change IRQ number in RHEL 6.4
Hi Guys
We have a HP Proliant DL380p Gen8 Server running with RHEL 6.4. This server also holds 2 Digium T1 Telephone cards. These cards share the IRQ 5 with other onboard PCI cards/devices. However while running some performance test with the card, we noticed that we are not able to obtain the maximum performance from these cards as they are sharing the IRQs with other devices. If we could assign a dedicated IRQ for these 2 devices, we would be able to obtain the best performance.
On that note, we contacted the Server manufacturer and the Card manufacturer. However Card manufacturer said there is nothing we can modify from their driver configuration to assign a different IRQ. And so the server manufacturer mentioned that we can't assign a dedicated IRQ for this device and need to check with OS side if they would all to do it.
That brings me here to this discussion forum. Could some experts on this field please advise if this is something that is doable from the RHEL OS? If it is possible, what is the right procedure to get it done?
Looking to hear from the experts here in this forum...
Regards
Jo
Attachments
Responses
Hello
There is no way to change the IRQ the way you want to do it as far as I am aware of.
Howewver, refer to this document from HP for tuning for low latency.
http://h20565.www2.hp.com/portal/site/hpsc/template.BINARYPORTLET/public/kb/docDisplay/resource.process/?spf_p.tpst=kbDocDisplay_ws_BI&spf_p.rid_kbDocDisplay=docDisplayResURL&javax.portlet.begCacheTok=com.vignette.cachetoken&spf_p.rst_kbDocDisplay=wsrp-resourceState%3DdocId%253Demr_na-c01804533-7%257CdocLocale%253D&javax.portlet.endCacheTok=com.vignette.cachetoken
There is a section in the document for RHEL to improve low latency performance. There is also a section showing you how to mask software IRQ's for CPU.
The cards you have are likely PCIE, they use MSI/MSIX so the way the interrupts are managed is different to hardware interrupts of older PCI cards. The sshared interrupts are not as significant performance degradation as cxompred to hardware IRQ's not using MSI/MSIX.
If you post a lspci here I can see if there is anything else I can help with. Poist the lspci and also the deatils of the driver.
Thanks
Laurence
Hi Jo,
How did you determine that the cards are sharing IRQs?
More information about your system and the specific cards from Digium would be helpful here.
As far as direct IRQs are concerned, the older PCI specification only had 4 IRQs shared among the entire PCI bus. In some PCI systems the IRQ line they used was directly wired to the PCI slot the card was in, others allowed the BIOS to configure the IRQ of each slot.
Most modern systems use MSI (Available in PCI since 2.2 and in PCI Express). http://en.wikipedia.org/wiki/Message_Signaled_Interrupts MSI-X allows cards to specify up to 2048 interrupts.
If your system BIOS does not allow you to configure the IRQ of the slot the cards are in and the cards do not support MSI, the only thing I can offer is to try to move the cards to a different PCI slot.
-Karl
Hi Jo,
I can't find that card on Digium's website, but I found it elsewhere, and it claims to be a PCI-Express card. In this case it should be using MSI/MSIX as Lawrence mentioned.
I'd like to see the lspci -vv and cat /proc/interrupts from the host to get a better idea of what's going on.
16: IR-IO-APIC-fasteoi uhci_hcd:usb3, hpilo, wct4xxp
58: IR-IO-APIC-fasteoi wct4xxp
So you have one card on IRQ 16 and one on IRQ 58, and it seems they don't support MSI/MSIX. Considering the other card isn't conflicting, it might be worth moving the card that is conflicting to another slot. That could solve the problem.
With IO-APIC there's nothing the driver/OS can do to rewire the IRQ, You may want to make sure that all processes using that IRQ are running on the same CPU. And the drivers all need to play nicely together as Jamie mentioned. You can see in your /proc/interrupts that IRQ 16 moves among CPU's whereas IRQ 58 stays on a single CPU.
You've already got good assistance above, but I thought it was worth pointing out a device's "willingness" to share interrupts with other devices is dictated by the device driver.
Specifically, this is done by setting a flag (SA_SHIRQ or IRQF_SHARED) when the driver runs request_irq(). The flag indicates that the device can share its IRQ with other devices. This is covered in further detail in Understanding the Linux Kernel and Linux Device Drivers.
The vendor's right that they can't modify the interrupt number assigned, but they may be able to modify the fact that the device is willing to share its IRQ with other devices.
(Edit: Just to clarify, I don't want to throw your vendor under the bus and accuse them of lying when they say they can't do anything. There might be a good reason they want their device to share interrupts. This just gives you the specific detail you could ask them to quantify using.)
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
