Chapter 1. Understanding InfiniBand and RDMA

InfiniBand refers to two distinct things:

  • The physical link-layer protocol for InfiniBand networks
  • The InfiniBand Verbs API, which is an implementation of the remote direct memory access (RDMA) technology

RDMA provides access to the memory from one computer to the memory of another computer without involving either computer’s operating system. This technology enables high-throughput and low-latency networking with low CPU utilization.

In a typical IP data transfer, when an application on one machine sends data to an application on another machine, the following happens on the receiving side:

  1. The kernel must receive the data.
  2. The kernel must determine that the data belongs to the application.
  3. The kernel wakes up the application.
  4. The kernel waits for the application to perform a system call into the kernel.
  5. The application copies the data from the kernel’s own internal memory space into the buffer provided by the application.

This process means that most network traffic is copied across the main memory of the system if the host adapter uses direct memory access (DMA), or otherwise at least twice. Additionally, the computer executes a number of context switches to switch between the kernel and application context. Both context switches can cause a high CPU load at high traffic rates and slow down other tasks.

RDMA communication bypasses the kernel intervention in the communication process, unlike the normal IP communication. This reduces the CPU overhead. The RDMA protocol enables the host adapter to know when a packet comes in from the network, which application should receive it, and where in the application’s memory space, the packet should be stored. Instead of sending the packet to the kernel to be processed and then copied into the user application’s memory, with InfiniBand, the host adapter places the contents of the packet directly in the application’s buffer. This process requires a separate API, the InfiniBand Verbs API, and applications must support this API before they can use RDMA.

Red Hat Enterprise Linux 8 supports both the InfiniBand hardware and the InfiniBand Verbs API. Additionally, Red Hat Enterprise Linux supports the following technologies that allow to use the InfiniBand Verbs API on non-InfiniBand hardware:

  • Internet Wide Area RDMA Protocol (iWARP): A network protocol that implements RDMA over IP networks.
  • RDMA over Converged Ethernet (RoCE), which is also known as InfiniBand over Ethernet (IBoE): A network protocol that implements RDMA over Ethernet networks.

Additional resources