NVMe drives failing to discover on Cisco UCS hardware

Issue

• NVMe drives failing to discover on Cisco UCS hardware

• The issue is only seen in Red Hat Enterprise Linux 6; not present in Red Hat Enterprise Linux 7.

• nvme_dev_start failure seen in /var/log/dmesg:

IRQ 0/nvme4q0: IRQF_DISABLED is not guaranteed on shared IRQs 
 [<ffffffffa008c510>] ? nvme_irq+0x0/0x60 [nvme]
 [<ffffffffa008c6bb>] ? queue_request_irq+0x3b/0x70 [nvme]
 [<ffffffffa00914bf>] ? nvme_dev_start+0x21f/0x6f0 [nvme]
 [<ffffffffa008c63d>] ? dma_free_coherent+0x4d/0x90 [nvme]
 [<ffffffffa0091b70>] ? nvme_async_probe+0x0/0x550 [nvme]
 [<ffffffffa0091ba3>] ? nvme_async_probe+0x33/0x550 [nvme]
 [<ffffffffa0091b70>] ? nvme_async_probe+0x0/0x550 [nvme]
nvme 0000:e2:00.0: PCI INT A: no GSI
nvme 0000:e2:00.0: setting latency timer to 64
nvme 0000:e3:00.0: PCI INT A: no GSI
nvme 0000:e3:00.0: setting latency timer to 64
Modules linked in: fnic(+) libfcoe libfc scsi_transport_fc scsi_tgt usb_storage sd_mod crc_t10dif sr_mod cdrom megaraid_sas(U) nvme xhci_hcd ahci wmi dm_mirror dm_region_hash dm_log dm_mod
 [<ffffffffa009138c>] ? nvme_dev_start+0xec/0x6f0 [nvme]
 [<ffffffffa009138c>] ? nvme_dev_start+0xec/0x6f0 [nvme]
 [<ffffffffa0091b70>] ? nvme_async_probe+0x0/0x550 [nvme]
 [<ffffffffa0091ba3>] ? nvme_async_probe+0x33/0x550 [nvme]
 [<ffffffffa0091b70>] ? nvme_async_probe+0x0/0x550 [nvme]
IRQ 0/nvme4q0: IRQF_DISABLED is not guaranteed on shared IRQs
Pid: 5169, comm: nvme Tainted: G        W  -- ------------    2.6.32-642.1.1.el6.x86_64 #1
 [<ffffffffa008c510>] ? nvme_irq+0x0/0x60 [nvme]
 [<ffffffffa008c6bb>] ? queue_request_irq+0x3b/0x70 [nvme]
 [<ffffffffa00914bf>] ? nvme_dev_start+0x21f/0x6f0 [nvme]
 [<ffffffffa0091a90>] ? nvme_reset_failed_dev+0x0/0xe0 [nvme]
 [<ffffffffa00919ae>] ? nvme_dev_resume+0x1e/0x80 [nvme]
 [<ffffffffa0091ac3>] ? nvme_reset_failed_dev+0x33/0xe0 [nvme]
 [<ffffffffa0091a90>] ? nvme_reset_failed_dev+0x0/0xe0 [nvme]

• Failure to assign memory in /var/log/messages:

pci 0000:e3:00.0: BAR 0: can't assign mem (size 0x4000)
pci 0000:e8:00.0: BAR 0: can't assign mem (size 0x4000)
<snip>
nvme 0000:e3:00.0: setting latency timer to 64
------------[ cut here ]------------
WARNING: at arch/x86/mm/ioremap.c:110 __ioremap_caller+0x33f/0x370() (Not tainted)
<snip>
nvme 0000:e3:00.0: Device not ready; aborting reset
nvme 0000:e8:00.0: Device not ready; aborting reset

• Missing devices:

rhel7.3 (5 drives discovered)
===========================
04:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller [144d:a802] (rev 01)
05:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller [144d:a802] (rev 01)
de:00.0 Non-Volatile memory controller [0108]: HGST, Inc. Ultrastar SN100 Series NVMe SSD [1c58:0003] (rev 05)
e1:00.0 Non-Volatile memory controller [0108]: HGST, Inc. Ultrastar SN100 Series NVMe SSD [1c58:0003] (rev 05) 
e3:00.0 Non-Volatile memory controller [0108]: HGST, Inc. Ultrastar SN100 Series NVMe SSD [1c58:0003] (rev 05)

rhel6.8 (4 drives discovered)
========================
04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller (rev 01)
05:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller (rev 01)
de:00.0 Non-Volatile memory controller: HGST, Inc. Ultrastar SN100 Series NVMe SSD (rev 05)
e1:00.0 Non-Volatile memory controller: HGST, Inc. Ultrastar SN100 Series NVMe SSD (rev 05)