NVMe over Fibre Channel (NVMe/FC) support and certification status.

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 7.5 or later
  • Broadcom/Emulex lpfc driver
    • in-box lpfc driver and storage capable of supporting NVMe over Fibre Channel
    • check with hardware vendor as to which model controllers support NVMe over Fibre Channel, which firmware release is required, and which RHEL releases they have been tested and certified on when using the in-box driver
  • Marvell/QLogic qla2xxx driver
    • in-box qla2xxx driver and storage capable of supporting NVMe over Fibre Channel
    • check with hardware vendor as to which model controllers support NVMe over Fibre Channel, which firmware release is required, and which RHEL releases they have been tested and certified on when using the in-box driver
  • IBM Coral Systems NVME over Fibre Channel target mode added in RHEL 8.1
    • check with hardware vendor for additional system level restrictions
    • check with hardware vendor as to which model controllers support NVMe over Fibre Channel, which firmware release is required, and which RHEL releases they have been tested and certified on when using the in-box driver

Issue

  • NVMe over Fibre Channel support and certification status.
  • Is NVMe over Fibre Channel (NVMe/FC) supported in RHEL?
  • Is the T11 "Fibre Channel - NVMe" specification supported in RHEL?
  • Is NVMe over Fabrics (NVMeoF/NVMe-oF) supported in Red Hat Enterprise Linux?
  • Why does my system have a tainted kernel when using the lpfc driver in 7.5?
  • Why does my system have a tainted kernel when using the qla2xxx driver in 7.5?
  • What are the nvme_fc and nvmet_fc modules and why do they taint the kernel in RHEL7?

Resolution

Support for NVMe over Fibre Channel was initially introduced in RHEL 7.5 as a Tech Preview within the lpfc driver in RHEL7.5. In 7.6, the feature within the lpfc driver was removed from the Tech Preview list so is fully supported in 7.6 and later. The qla2xxx driver with NVM over Fibre Channel support was added in 7.7 as "Tech Preview".

Within RHEL 8.0 and later both the lpfc and qla2xxx drivers are fully supported with the NVMe over Fibre Channel feature enabled.

If you have any further questions regarding the supportability status of the Fibre Channel HBA or the NVMe over Fibre Channel capabilities of this system, please contact the hardware vendor.

What is NVMe over Fibre Channel?

Most people are familiar with attaching SCSI devices over Fibre Channel. The Fibre Channel is a transport protocol upon which different upper layer protocols (ULPs) are mapped. For example, for SCSI the SCSI-FCP standard specification defines how SCSI commands , data and status are mapped on top of the Fibre Channel transport layer (this is a simplification, but a reasonable one). Other upper layer protocols exist for mapping FICON to Fibre Channel for example, among others.

NVMe over Fibre Channel is another such ULPs. This protocol is variously referred to as NVMEoF, NVMe-oF, NVMe/FC, FC-NVMe among others.

  • A generic term of "NVMe over Fabrics" (NVMeoF/NVMe-oF/NVMf) is used to refer to mapping of NVMe onto any storage fabric, not Fibre Channel specifically.
  • While FC-NVMe refers to the actual T11 "Fibre Channel - NVMe" FC-4 layer mapping specification. This specification uses services defined in the FC framing and signal inferface level (FC-FS-5) to pass NVMe command, data, and status between a host and storage subsystem.
  • NVMe/FC is used as a short hand for an implementation of FC-NVMe, much the same as NVMe/RDMA and NVMe/TCP are implementations of NVMe onto those transport protocols.

In summary, NVMe over FC (NVMe/FC) is an implementation of the T11 "Fibre Channel - NVMe" standard specification that supports passing NVMe commands, data, status over Fibre Channel.

Feature Availability Matrix

The following matrix shows which drivers were available with the NVME/FC feature, in which RHEL releases, and whether it was a "Tech Preview" or fully supported driver (Tech Preview='N' means available and fully supported).

Driver RHEL Mode[2] Tech Preview?[1]
lpfc 8.0+ initiator N
lpfc 7.7 + initiator N
lpfc 7.6 initiator N
lpfc 7.5 initiator Y
qla2xxx 8.0 initiator N
qla2xxx 7.7 initiator Y
qla2xxx 7.6 initiator ---
qla2xxx 7.5 initiator ---

  • Notes

    • [1] "Technology Preview" features are not fully supported under Red Hat Subscription Level Agreements, may not be functionally complete, and are not intended for production use. Driver versions that are no longer marked Tech Preview are fully supported, with any exceptions as outlined in the appropriate release notes (typically not supported is multipath, booting, kdump, ...).

      • Use of a tech preview driver will result in the kernel being tainted as 'TECH_PREVIEW' and the following will be logged within messages:

        TECH PREVIEW: NVMe over FC may not be fully supported.
Please review provided documentation for limitations.

    • [2] In initiator mode, commands are sent through the HBA to external storage. This is the normal operating mode for most HBA. Some of the HBAs can operate in target (server) mode. When in target mode the HBA acts as a storage port and receives NVMe commands from other hosts. HBAs operating in target mode are not supported for NVMe/FC operations: "Tech Preview" only in RHEL 7.

    • [3] When either the lpfc or qla2xxx driver is loaded with the NMVe/FC feature enabled, it causes the nvme_fc (and nvmet_fc for lpfc) modules to also be loaded in order to support this feature. They provide generic initiator and target support, respectively. These modules are marked Tech Preview, and as such, they taint the kernel. It is not currently possible to blacklist these modules and still have the device fully functional even for non-NVMe storage. It is not currently possible to blacklist these modules and still have the device fully functional even for non-NVMe storage. As long as customers do not enable the NVMe/FC feature within these drivers, it will not affect the supportability of either the driver and associated devices. In other words, without this feature enabled, the driver is in fully supported state and is not considered "Tech Preview".

      {show which other modules will be loaded when the driver is loaded...}
# modinfo lpfc    | grep depends
  depends:        nvme-fc,scsi_transport_fc,nvmet-fc
# modinfo qla2xxx | grep depends
  depends:        nvme-fc,scsi_transport_fc

Enabling NVMe/FC

The two drivers available for NVMe/FC support may require additional configuration options to be set in order to enable the feature. Specifying driver options is of the general format of:

options <driver-name> <option-name1=value1> <option-name2=value2> ... <option-nameN=valueN>

The following instructions highlight adding the necessary option-name:option-value to the driver's configuration file. If the file does not exist, you will need to create it. If the options <driver-name> does not exist you will need to add it.

  • LPFC: To enable NVMe/FC feature within the lpfc driver, edit the /etc/modprobe.d/lpfc.conf file and add the following option:
    • For all kernel versions (setting the value to 1 will turn off NVMe/FC feature and only support FCP-SCSI, it is set to 3 (on) by default in most versions):
options lpfc {other options if already present}  lpfc_enable_fc4_type=3
  • QLA2XXX: To enable NVMe/FC feature within the qla2xxx driver, edit the /etc/modprobe.d/qla2xxx.conf file and add the following option:
    • For all kernel versions: (setting the value to 0 will turn off NVMe/FC feature, it is set to 1 (on) by default in most versions)
options qla2xxx {other options if already present}  ql2xnvmeenable=1

Certification

Fibre Channel HBAs that have the NVMe/FC feature within their driver(s) but are listed as "Tech Preview" are not tested as part of Red Hat's Hardware Certification program.

If this article is linked to a system in the Certification Catalog, the NVMe/FC functionality was not tested in those cases the driver was "Tech Preview". As with all "Tech Preview" features, such supplied in-box drivers should not be used in a production environment.

Applicable Release Notes

Since this is still an evolving technology, check the release notes for any RHEL version 7.5 or later for specific details on NVMe/FC support.


  • RHEL 8.0 5.1.12. File systems and storage, added full support in qla2xxx (lpfc already had full support in RHEL 7.6 and later).

    The NVMe over Fibre Channel (NVMe/FC) transport type is now fully supported in Initiator mode when used with Broadcom Emulex and Marvell Qlogic Fibre Channel 32Gbit adapters that feature NVMe support.
    • See the additional restrictions section for the lpfc and qla2xxx driver with regards to NVMe over FC within 8.0 release.
      • Multipath is not supported with NVMe/FC.
      • kdump is not supported with NVMe/FC.
      • Booting from Storage Area Network (SAN) NVMe/FC is not supported.
      • NVMe clustering is not supported with NVMe/FC.

    • Also in RHEL 8.0, support for NVMe over RDMA (NVMe/RDMA) was added.

      In Red Hat Enterprise Linux (RHEL) 8, Nonvolatile Memory Express (NVMe) over Remote Direct Memory Access (RDMA) supports    Infiniband, RoCEv2, and iWARP only in initiator mode.  Note that Multipath is supported in failover mode only.
      • Listed restrictions on NVMe/RDMA
        • Kdump is not supported with NVMe/RDMA.
        • Booting from a NVMe /RDMA device is not supported.




  • RHEL7.5 Chapter 44. Kernel, introduced NVME/FC support in lpfc driver, "Tech Preview" only.

    The NVMe over Fibre Channel transport type is now available as a Technology Preview. 
:
In Red Hat Enterprise Linux 7.5, NVMe over Fibre Channel is available only with Broadcom 32Gbit adapters, which use the lpfc driver.
    • Feature of NVMe/FC first introduced as Tech Preview in 7.5 but only in lpfc driver for specific Broadcom/Emulex Fibre Channel adapters.
    • To configure an NVMe target, use the nvmetcli utility.


Notes

Diagnostic Steps

  • To confirm you are not enabling NVME/FC

    • With the lpfc driver ensure that the following parameter is not set to '3' (value of 1 is off, SCSI device support only, it is set to '3' by default)..

      # modinfo lpfc | grep lpfc_enable_fc4
parm:           lpfc_enable_fc4_type:Enable FC4 Protocol support - FCP / NVME (uint)
# 
 grep -Hv "zz" /sys/module/lpfc/parameters/lpfc_enable_fc4_type
/sys/module/lpfc/parameters/lpfc_enable_fc4_type:3

      • Defined values within the drive:

        #define LPFC_ENABLE_FCP  1      <= SCSI-FCP only
#define LPFC_ENABLE_NVME 2      <= NVMe/FC only   
#define LPFC_ENABLE_BOTH 3      <= Both

    • With qla2xxx driver ensure the following parameter is not set to '1' (value 0 is off, it is enabled (1) by default)

      # modinfo qla2xxx | grep nvme
depends:        nvme-fc,scsi_transport_fc
parm:           ql2xnvmeenable:Enables NVME support. 0 - no NVMe.  Default is Y (int)
# grep -Hv "zz" /sys/module/qla2xxx/parameters/ql2xnvmeenable
/sys/module/qla2xxx/parameters/ql2xnvmeenable:1

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments