Chapter 29. Overview of NVMe over fabric devices

Non-volatile Memory Express (NVMe) is an interface that allows host software utility to communicate with solid state drives. Use the following types of fabric transport to configure NVMe over fabric devices:
When using FC and RDMA, the solid-state drive does not have to be local to your system; it can be configured remotely through a FC or RDMA controller.

29.1. NVMe over fabrics using RDMA

The following sections describe how to deploy an NVMe over RDMA (NVMe/RDMA) initiator configuration.

29.1.1. Configuring an NVMe over RDMA client

Use this procedure to configure an NVMe/RDMA client using the NVMe management command line interface (nvme-cli).
  1. Install the nvme-cli package:
    # yum install nvme-cli
  2. Load the nvme-rdma module if it is not loaded:
    # modprobe nvme-rdma
  3. Discover available subsystems on the NVMe target:
    # nvme discover -t rdma -a 172.31.0.202 -s 4420
    
    Discovery Log Number of Records 1, Generation counter 2
    =====Discovery Log Entry 0======
    trtype:  rdma
    adrfam:  ipv4
    subtype: nvme subsystem
    treq:    not specified, sq flow control disable supported
    portid:  1
    trsvcid: 4420
    subnqn:  testnqn
    traddr:  172.31.0.202
    rdma_prtype: not specified
    rdma_qptype: connected
    rdma_cms:    rdma-cm
    rdma_pkey: 0x0000
    
  4. Connect to the discovered subsystems:
    # nvme connect -t rdma -n testnqn -a 172.31.0.202 -s 4420
    
    # lsblk
    
    NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
    sda                            8:0    0 465.8G  0 disk
    ├─sda1                         8:1    0     1G  0 part /boot
    └─sda2                         8:2    0 464.8G  0 part
      ├─rhel_rdma--virt--03-root 253:0    0    50G  0 lvm  /
      ├─rhel_rdma--virt--03-swap 253:1    0     4G  0 lvm  [SWAP]
      └─rhel_rdma--virt--03-home 253:2    0 410.8G  0 lvm  /home
    nvme0n1
    
    # cat /sys/class/nvme/nvme0/transport
    
    rdma
    
    Replace testnqn with the NVMe subsystem name.
    Replace 172.31.0.202 with the target IP address.
    Replace 4420 with the port number.
  5. List the NVMe devices that are currently connected:
    # nvme list
  6. Optional: Disconnect from the target:
    # nvme disconnect -n testnqn
    
    NQN:testnqn disconnected 1 controller(s)
    
    # lsblk
    
    NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
    sda                            8:0    0 465.8G  0 disk
    ├─sda1                         8:1    0     1G  0 part /boot
    └─sda2                         8:2    0 464.8G  0 part
    ├─rhel_rdma--virt--03-root 253:0    0    50G  0 lvm  /
    ├─rhel_rdma--virt--03-swap 253:1    0     4G  0 lvm  [SWAP]
    └─rhel_rdma--virt--03-home 253:2    0 410.8G  0 lvm  /home
    

Additional resources