Chapter 13. Configuring NVMe over fabrics using NVMe/TCP

In an Non-volatile Memory Express™ (NVMe™) over TCP (NVMe/TCP) setup, the host mode is fully supported and the controller setup is not supported.

As a system administrator, complete the tasks in the following sections to deploy the NVMe/TCP setup:

Note

In Red Hat Enterprise Linux 9, the native NVMe multipathing is enabled by default. Enabling DM multipathing is not supported with NVMe/TCP.

13.1. Overview of NVMe over fabric devices

Non-volatile Memory Express™ (NVMe™) is an interface that allows host software utility to communicate with solid state drives.

Use the following types of fabric transport to configure NVMe over fabric devices:

NVMe over Remote Direct Memory Access (NVMe/RDMA)
For information about how to configure NVMe™/RDMA, see Configuring NVMe over fabrics using NVMe/RDMA.
NVMe over Fibre Channel (NVMe/FC)
For information about how to configure NVMe™/FC, see Configuring NVMe over fabrics using NVMe/FC.
NVMe over TCP (NVMe/TCP)
For information about how to configure NVMe/FC, see Configuring NVMe over fabrics using NVMe/TCP.

When using NVMe over fabrics, the solid-state drive does not have to be local to your system; it can be configured remotely through a NVMe over fabrics devices.

13.2. Configuring an NVMe/TCP host

Use the Non-volatile Memory Express™ (NVMe™) management command line interface (nvme-cli) tool to configure an NVMe/TCP host.

Procedure

  1. Install the nvme-cli tool: 

    # dnf install nvme-cli

    This tool creates the hostnqn file in the /etc/nvme/ directory, which identifies the NVMe host.

  2. Find the nvme hostid and hostnqn: 

    # cat /etc/nvme/hostnqn
nqn.2014-08.org.nvmexpress:uuid:8ae2b12c-3d28-4458-83e3-658e571ed4b8

# cat /etc/nvme/hostid
09e2ce17-ccc9-412d-8dcf-2b0a1d581ee3

    Use the hostid and hostnqn values to configure the NVMe/TCP controller.

  3. Check the status of the controller: 

    # nmcli device show ens6
GENERAL.DEVICE:                         ens6
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         52:57:02:12:02:02
GENERAL.MTU:                            1500
GENERAL.STATE:                          30 (disconnected)
GENERAL.CONNECTION:                     --
GENERAL.CON-PATH:                       --
WIRED-PROPERTIES.CARRIER:               on

  4. Configure the host network for a newly installed Ethernet controller with a static IP address: 

    # nmcli connection add con-name ens6 ifname ens6 type ethernet ip4 192.168.101.154/24 gw4 192.168.101.1

    Here, replace 192.168.101.154 with the host IP address. 

    # nmcli connection mod ens6 ipv4.method manual
# nmcli connection up ens6

    Since a new network is created to connect the NVMe/TCP host to the NVMe/TCP controller, execute this step on the controller too.

Verification

  • Verify if the newly created host network works correctly: 

    # nmcli device show ens6
GENERAL.DEVICE:                         ens6
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         52:57:02:12:02:02
GENERAL.MTU:                            1500
GENERAL.STATE:                          100 (connected)
GENERAL.CONNECTION:                     ens6
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/5
WIRED-PROPERTIES.CARRIER:               on
IP4.ADDRESS[1]:                         192.168.101.154/24
IP4.GATEWAY:                            192.168.101.1
IP4.ROUTE[1]:                           dst = 192.168.101.0/24, nh = 0.0.0.0, mt = 101
IP4.ROUTE[2]:                           dst = 192.168.1.1/32, nh = 0.0.0.0, mt = 101
IP4.ROUTE[3]:                           dst = 0.0.0.0/0, nh = 192.168.1.1, mt = 101
IP6.ADDRESS[1]:                         fe80::27ce:dde1:620:996c/64
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = fe80::/64, nh = ::, mt = 101

Additional resources

  • The nvme(1) man page

13.3. Connecting the NVMe/TCP host to the NVMe/TCP controller

Connect the NVMe™ over TCP (NVMe/TCP) host to the NVMe/TCP controller system to verify that the NVMe/TCP host can now access the namespace.

Note

The NVMe/TCP controller (nvmet_tcp) module is not supported.

Prerequisites

  • You have configured an NVMe/TCP host. For more information, see Configuring an NVMe/TCP host.
  • You have configured an NVMe/TCP controller using external storage software and the network is configured on the controller. In this procedure, 192.168.101.55 is the IP address of NVMe/TCP controller.

Procedure

  1. Load the nvme_tcp module if not already: 

    # modprobe nvme_tcp

  2. Discover the available subsystems on the NVMe controller: 

    # nvme discover --transport=tcp --traddr=192.168.101.55 --host-traddr=192.168.101.154 --trsvcid=8009

Discovery Log Number of Records 2, Generation counter 7
=====Discovery Log Entry 0======
trtype:  tcp
adrfam:  ipv4
subtype: current discovery subsystem
treq:	not specified, sq flow control disable supported
portid:  2
trsvcid: 8009
subnqn:  nqn.2014-08.org.nvmexpress.discovery
traddr:  192.168.101.55
eflags:  not specified
sectype: none
=====Discovery Log Entry 1======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:	not specified, sq flow control disable supported
portid:  2
trsvcid: 8009
subnqn:  nqn.2014-08.org.nvmexpress:uuid:0c468c4d-a385-47e0-8299-6e95051277db
traddr:  192.168.101.55
eflags:  not specified
sectype: none

    Here, 192.168.101.55 is the NVMe/TCP controller IP address and 192.168.101.154 is the NVMe/TCP host IP address.

  3. Configure the /etc/nvme/discovery.conf file to add the parameters used in the nvme discover command : 

    # echo "--transport=tcp --traddr=192.168.101.55 --host-traddr=192.168.101.154 --trsvcid=8009" >> /etc/nvme/discovery.conf

  4. Connect the NVMe/TCP host to the controller system: 

    # nvme connect-all

Verification

  • Verify that the NVMe/TCP host can access the namespace: 

    # nvme list-subsys

nvme-subsys3 - NQN=nqn.2014-08.org.nvmexpress:uuid:0c468c4d-a385-47e0-8299-6e95051277db
\
 +- nvme3 tcp traddr=192.168.101.55,trsvcid=8009,host_traddr=192.168.101.154 live optimized

# nvme list
Node              	Generic           	SN               	Model                                	Namespace Usage                  	Format       	FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme3n1      	/dev/ng3n1        	d93a63d394d043ab4b74 Linux                                    1          21.47  GB /  21.47  GB    512   B +  0 B   5.18.5-2

Additional resources

  • The nvme(1) man page