InfiniBand RDMA and RoCE Explained: Protocols, Messages, and Network Architecture

As high-performance computing, AI training, and distributed systems continue to scale, inter-node communication efficiency has become a crucial factor affecting overall performance. RDMA effectively reduces communication latency and improves bandwidth utilization by bypassing the operating system kernel and reducing data copying.

This article will focus on the fundamental principles of RDMA, systematically outlining the technical differences between InfiniBand RDMA and RoCE. It will compare and analyze message formats, protocol layers, and network construction methods to help readers understand the differences between the two more clearly in different application scenarios.

What is RDMA (Remote Direct Memory Access)?

RDMA (Remote Direct Memory Access) is a technology used to improve the data transmission efficiency of traditional Ethernet networks. It bypasses the host operating system kernel and intermediate caches to achieve direct memory data transmission across nodes, thereby reducing server-side data processing overhead. This technology relies on RDMA-enabled network interface cards (NICs) and switching devices and is widely used in high-performance scenarios such as large-scale scientific computing and model training and inference.

RDMA Core Features

Main RDMA Solutions: InfiniBand and RoCE

Currently, there are three main RDMA-based solutions: InfiniBand, RoCE, and iWARP. For a comparison of these three, please refer to the article: What is RDMA? RoCE vs. InfiniBand vs. iWARP Difference. InfiniBand and RoCE are the mainstream solutions, and we will focus on these two technologies below.

• InfiniBand-based RDMA: InfiniBand natively supports RDMA communication. Through a channel-based point-to-point message queue forwarding model, it allows applications to directly access remote memory without kernel and CPU intervention. However, InfiniBand networks require dedicated hardware for networking, resulting in higher costs. However, its hardware architecture ensures high efficiency, low latency, and reliability in data transmission.
• RoCE (RDMA over Converged Ethernet): This is a technology that implements RDMA over traditional Ethernet, divided into RoCE v1 and RoCE v2. RoCE v1 is implemented based on the Ethernet link layer and relies on Priority Flow Control (PFC) to ensure transmission reliability. It does not support cross-layer 3 networks. RoCE v2 is implemented based on the Ethernet UDP/IP protocol, can run on layer 3 networks, supports cross-segment communication, and has better network scalability, thus it is widely used in intelligent computing data centers.

Differences between InfiniBand and RoCE

What are the Differences in Message Formats Between RoCE and InfiniBand?

InfiniBand Message Format

InfiniBand has a complete and independent protocol stack. Taking the RC (Reliable Connection) type as an example, its message consists of the following parts:

• LRH (Local Routing Header): Used for switching and forwarding within a subnet, operating at the data link layer. This header has a built-in lossless transmission mechanism, ensuring data reliability at the link layer through a credit-based flow control method.
• GRH (Global Routing Header): Used for cross-subnet communication, belonging to the network layer header. GRH and LRH together constitute the InfiniBand routing mechanism, introducing a 128-bit GID (Global Identifier) ​​similar to IPv6 to support cross-subnet addressing.
• BTH (Base Transport Header): The BTH is the basic header of the InfiniBand transport layer (L4), used to describe the transmission control information of data packets. It is a core component of the IB transport mechanism and a key control unit for RDMA operations.
• ETH (RDMA Extended Transport Header): This extended header supplements the BTH with the control information required for RDMA, including key fields such as remote virtual address, access key, and data length.
• Payload: Carries the actual transmitted RDMA data or message content.
• ICRC (Invariant CRC): Used to verify the data content from the GRH (if present) or BTH to the end of the payload. This checksum remains unchanged throughout the network forwarding process.
• VCRC (Variant CRC): Covers the entire message and is recalculated after each switching node, used to detect link-level transmission errors.

InfiniBand networks rely on a Subnet Manager (SM) for address allocation and routing management. The network can only be used after the SM is functioning correctly.

RoCE Message Format

RoCE has two versions with significantly different message formats.

Get NADDOD’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

RoCEv1:

• Ethernet L2 Header: Uses the standard Ethernet L2 Header.
• EtherType (0x8915): This value identifies RoCEv1 data frames.
• GRH: Inherited from the InfiniBand protocol. Since Ethernet already handles routing, this header has limited practical use in RoCEv1, but it is retained for compatibility with IB semantics.
• BTH / Payload / ICRC: These three parts are consistent with InfiniBand messages and are core to the continuation of RDMA semantics.
• FCS (Frame Check Sequence): The standard Ethernet frame check field used for link layer error detection.

RoCEv2 introduces a three-layer network encapsulation layer on top of RoCEv1, adding IP and UDP headers to address the limitation of scalability.

• Ethernet L2 Header: Uses the standard Ethernet L2 header.
• EtherType (0x0800): Indicates that the packet carries an IP data packet.
• IP Header: Can be IPv4 or IPv6, enabling RoCEv2 packets to have three-layer routing capabilities and forward across multiple subnets.
• UDP Header: Uses standard UDP encapsulation, with a fixed destination port of 4791, used to identify RDMA traffic.
• BTH / Payload / ICRC: Consistent with InfiniBand packets; the core semantics of RDMA remain unchanged.
• FCS: Ethernet Link Layer Checksum field.

Whether using InfiniBand or RoCE, the Base Transport Header (BTH) remains the core control layer for RDMA operations, carrying critical information such as Opcode, PSN, and QP. ICRC ensures the integrity of this critical information during end-to-end transmission. The main differences between the two are as follows:

• InfiniBand: Designed specifically for RDMA scenarios from the physical layer to the transport layer, its protocol structure is simple and direct, providing optimal transmission efficiency, but it has specific requirements for network devices and infrastructure.
• RoCE: Designed to implement RDMA capabilities in Ethernet environments. RoCEv2 uses standard UDP/IP encapsulation for better network compatibility and can be deployed on existing data center network architectures.

RoCE and InfiniBand Network Layer Comparison

InfiniBand natively supports RDMA, and its protocol architecture, from the link layer to the transport layer, is built around RDMA scenarios, and it includes end-to-end verification mechanisms such as the Base Transport Header (BTH) and ICRC.

RoCE encapsulates RDMA on top of Ethernet. It wraps RDMA packets on top of Ethernet and UDP/IP protocols, embedding the InfiniBand transport layer payload (data) within them, thus inheriting the core transport mechanism of InfiniBand in an Ethernet environment.

This encapsulation method gives RoCE greater network compatibility, enabling it to run on general Ethernet infrastructure. However, it also adds additional overhead and complexity. For example, RoCE requires a lossless Ethernet environment to ensure reliable transport. In contrast, InfiniBand’s link layer is inherently reliable, a capability built on dedicated hardware and higher deployment costs.

From a protocol structure perspective, InfiniBand and RoCE are consistent at the transport layer and above, with the main differences concentrated in the physical layer, data link layer, and network layer. A detailed comparison is shown in the diagram below.

How to Build an InfiniBand RDMA and RoCE Network Environment?

InfiniBand RDMA

Building an InfiniBand (IB) network relies on several key components, primarily including the Host Channel Adapter (HCA), InfiniBand switch, Subnet Manager (SM), and InfiniBand optical modules.

• HCA (Host Channel Adapter): Also known as the InfiniBand network card, it is the interface through which servers access the IB network, handling RDMA data transmission operations and serving as the starting point for node-network communication.
• InfiniBand Switch: Located in the network core layer, it connects the HCAs of various servers and switching devices at different levels, enabling high-speed forwarding and switching of data packets.
• InfiniBand Optical Modules and Cables: Used to connect the HCA to the switch or between switches for data transmission. With technological advancements, InfiniBand connectivity products have evolved from 200G HDR and 400G/800G NDR to the current 1.6T XDR.
• Subnet Manager (SM): As the control core of the InfiniBand network, it is responsible for subnet initialization and operational management, including discovering network devices and topology, and assigning Local Identifiers (LIDs) to each HCA. In actual deployments, this functionality is often implemented through UFM or OpenSM.
• LID (Local Identifier): A 16-bit unique address used within the subnet for packet routing and forwarding, uniformly allocated and managed by the SM.
• GID (Global Identifier): A 128-bit globally unique identifier, similar in format to IPv6 addresses, primarily used for cross-subnet communication and global addressing scenarios.

Roce

Unlike InfiniBand, which relies on centralized management components (such as the Subnet Manager SM), RoCE v2 is built on Ethernet and employs a decentralized distributed architecture. The network consists of RoCE v2-enabled network interface cards (NICs) and Ethernet switches, typically used in two-tier network architectures within data centers.

Similar to InfiniBand, RoCE networks require several core components to work together:

• Network Adapters: Major manufacturers include NVIDIA and Broadcom, with products primarily consisting of PCIe interface RDMA cards. NVIDIA’s ConnectX-9 super NIC provides 1×800G transmission capability per port in Ethernet mode.
• Ethernet Switches: NVIDIA, Cisco, HPE, Arista, and other manufacturers offer Ethernet switching devices for RoCE scenarios, used in conjunction with RoCE NICs to achieve end-to-end RDMA communication. High-performance switches typically rely on advanced forwarding chip solutions. Broadcom’s Tomahawk series is widely used due to its high throughput and high bandwidth characteristics, with the latest Tomahawk 6 chip supporting Ethernet switching capabilities up to 102.4 Tbps.
• Ethernet Optical Modules and Cables: Since RoCE operates based on the Ethernet protocol, standard Ethernet optical modules and cables can be used directly, effectively reducing overall hardware deployment costs while ensuring performance.

Conclusion

Both InfiniBand and RoCE are based on RDMA, but they differ in their implementation paths. InfiniBand uses a dedicated network and centralized management, offering stable and predictable performance. RoCE, on the other hand, implements RDMA over Ethernet, especially RoCE v2, which offers advantages in compatibility and deployment flexibility.