Red Hat Training

A Red Hat training course is available for Red Hat Enterprise Linux

Chapter 29. Overview of NVMe over fabric devices

Non-volatile Memory Express (NVMe) is an interface that allows host software utility to communicate with solid state drives. Use the following types of fabric transport to configure NVMe over fabric devices:
When using FC and RDMA, the solid-state drive does not have to be local to your system; it can be configured remotely through a FC or RDMA controller.

29.1. NVMe over fabrics using RDMA

The following sections describe how to deploy an NVMe over RDMA (NVMe/RDMA) initiator configuration.

29.1.1. Configuring an NVMe over RDMA client

Use this procedure to configure an NVMe/RDMA client using the NVMe management command line interface (nvme-cli).
  1. Install the nvme-cli package: 
    # yum install nvme-cli
  2. Load the nvme-rdma module if it is not loaded: 
    # modprobe nvme-rdma
  3. Discover available subsystems on the NVMe target: 
    # nvme discover -t rdma -a 172.31.0.202 -s 4420

Discovery Log Number of Records 1, Generation counter 2
=====Discovery Log Entry 0======
trtype:  rdma
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified, sq flow control disable supported
portid:  1
trsvcid: 4420
subnqn:  testnqn
traddr:  172.31.0.202
rdma_prtype: not specified
rdma_qptype: connected
rdma_cms:    rdma-cm
rdma_pkey: 0x0000
  4. Connect to the discovered subsystems: 
    # nvme connect -t rdma -n testnqn -a 172.31.0.202 -s 4420

# lsblk

NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0 465.8G  0 disk
├─sda1                         8:1    0     1G  0 part /boot
└─sda2                         8:2    0 464.8G  0 part
  ├─rhel_rdma--virt--03-root 253:0    0    50G  0 lvm  /
  ├─rhel_rdma--virt--03-swap 253:1    0     4G  0 lvm  [SWAP]
  └─rhel_rdma--virt--03-home 253:2    0 410.8G  0 lvm  /home
nvme0n1

# cat /sys/class/nvme/nvme0/transport

rdma
    Replace testnqn with the NVMe subsystem name.
    Replace 172.31.0.202 with the target IP address.
    Replace 4420 with the port number.
  5. List the NVMe devices that are currently connected: 
    # nvme list
  6. Optional: Disconnect from the target: 
    # nvme disconnect -n testnqn

NQN:testnqn disconnected 1 controller(s)

# lsblk

NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0 465.8G  0 disk
├─sda1                         8:1    0     1G  0 part /boot
└─sda2                         8:2    0 464.8G  0 part
├─rhel_rdma--virt--03-root 253:0    0    50G  0 lvm  /
├─rhel_rdma--virt--03-swap 253:1    0     4G  0 lvm  [SWAP]
└─rhel_rdma--virt--03-home 253:2    0 410.8G  0 lvm  /home

Additional resources