Chapter 3. Configuring RoCE
This section explains background information about RDMA over Converged Ethernet (RoCE), as well as how to change the default RoCE version.
Note that there are different vendors, such as Mellanox, Broadcom, and QLogic, who provide RoCE hardware.
3.1. Overview of RoCE protocol versions
RoCE is a network protocol that enables remote direct memory access (RDMA) over Ethernet.
The following are the different RoCE versions:
- RoCE v1
The RoCE version 1 protocol is an Ethernet link layer protocol with ethertype
0x8915that enables the communication between any two hosts in the same Ethernet broadcast domain.
- RoCE v2
The RoCE version 2 protocol exists on the top of either the UDP over IPv4 or the UDP over IPv6 protocol. For RoCE v2, the UDP destination port number is
The RDMA_CM sets up a reliable connection between a client and a server for transferring data. RDMA_CM provides an RDMA transport-neutral interface for establishing connections. The communication uses a specific RDMA device and message-based data transfers.
Using different versions like RoCE v2 on the client and RoCE v1 on the server is not supported. In such a case, configure both the server and client to communicate over RoCE v1.
RoCE v1 works at the Data Link layer (Layer 2) and only supports the communication of two machines in the same network. By default, RoCE v2 is available. It works at the Network Layer (Layer 3). RoCE v2 supports packets routing that provides a connection with multiple Ethernet.
3.2. Temporarily changing the default RoCE version
Using the RoCE v2 protocol on the client and RoCE v1 on the server is not supported. If the hardware in your server only supports RoCE v1, configure your clients to communicate with the server using RoCE v1. This section describes how to enforce RoCE v1 on the client that uses the
mlx5_0 driver for the Mellanox ConnectX-5 Infiniband device.
Note that the changes described in this section are only temporary until you reboot the host.
- The client uses an InfiniBand device with RoCE v2 protocol
- The server uses an InfiniBand device that only supports RoCE v1
# mkdir /sys/kernel/config/rdma_cm/mlx5_0/
Display the default RoCE mode:
# cat /sys/kernel/config/rdma_cm/mlx5_0/ports/1/default_roce_mode RoCE v2
Change the default RoCE mode to version 1:
# echo "IB/RoCE v1" > /sys/kernel/config/rdma_cm/mlx5_0/ports/1/default_roce_mode