High-Performance Computing (HPC) networking — Demystifying InfiniBand, RoCE, and Mellanox Cards

In the era of explosive AI growth and massive datasets, the bottleneck in the data center is rarely the processor's speed. It’s about how fast it can get the data it needs to think about.

When you are training a large language model or running high-frequency trading algorithms, standard networking — the kind that connects your laptop to the internet — just doesn’t cut it. The latency is too high, and the CPU overhead is too intense.

Enter the world of High-Performance Computing (HPC) networking. If you’ve stepped into this arena, you’ve likely been hit with a barrage of acronyms: InfiniBand, RoCE, and the ubiquitous “Mellanox card.”

What do they mean, and more importantly, how do these three distinct concepts relate to each other? Let’s break it down

The Secret Sauce: RDMA

Before we define the protocols, we need to understand the magic trick that enables high-speed networking: RDMA (Remote Direct Memory Access). RDMA (Remote Direct Memory Access) is a networking technology that allows one computer to read and write another computer's memory directly, without involving either CPU or operating system, resulting in extremely low latency and very high throughput.

To understand RDMA, you first need to look at why standard networking (like standard TCP/IP) is “slow” for high-performance tasks:

1. Buffer Copying: When App A sends data to App B over TCP/IP, the data is copied from the Application space → Kernel space → Network Card buffer. The reverse happens on the receiving end. Every copy takes time.
2. Context Switching: The CPU must pause the application, switch to “Kernel Mode” to handle the network traffic, and then switch back.
3. CPU Overhead: The CPU is heavily involved in packaging data, calculating checksums, and managing transmission, thereby stealing cycles that could be used for computation instead.

RDMA bypasses this entire software stack.

• Kernel Bypass: The application communicates directly with the Network Interface Card (NIC). The OS kernel is not involved in the data path.
• Zero Copy: Data is sent directly from the memory of one application into the memory of the remote application. There are no intermediate copies to kernel buffers.
• CPU Offload: The actual movement of data is handled by the silicon on the network card (often called an HCA (Host Channel Adapter) or RNIC (RDMA-enabled Network Interface Card) — specialized, high-speed network adapters used to facilitate low-latency, high-bandwidth data transfers between compute nodes bypassing the operating system’s CPU for direct memory access (RDMA)), leaving the CPU free to do other work

It’s like having movers who have a key to the destination house and put the boxes straight into the correct rooms without bothering the homeowner.

While RDMA started with specialized hardware, it now runs over different types of cables:

1. InfiniBand (IB): The original standard for RDMA. It requires specialized switches and cables. It is the dominant standard in supercomputers.
2. RoCE (RDMA over Converged Ethernet): Pronounced “Rocky.” It allows you to run RDMA over standard Ethernet networks, provided the switches and NICs support it. This is popular in modern enterprise data centers.
3. iWARP: A protocol that runs RDMA over standard TCP/IP, though it is generally considered less efficient than RoCE or InfiniBand.

RDMA allows a NIC (like InfiniBand, RoCE, or iWARP) to:

• Transfer data directly between application memory buffers on two machines
• Bypass the kernel, system calls, and context switches
• Avoid CPU involvement on both sides
• Perform zero‑copy data movement
• Deliver microsecond‑level latency and line‑rate throughput

How RDMA Works (Conceptually)

To understand how RDMA works “under the hood,” we have to look at the interaction between the application, the Operating System, and the hardware (the Network Interface Card, or NIC).

Before any data transfer can happen, the application must “introduce” a specific region of its memory to the NIC. This is a critical step called Memory Registration. The OS normally swaps memory pages to disk or moves them around (especially in Java with Garbage Collection). RDMA requires the memory to stay fixed at a specific physical address. Registration “pins” this memory so the OS cannot move it. The NIC is given a translation table (Virtual Address →Physical Address) so it knows exactly where to read/write data on the RAM stick without asking the CPU

Note for Java Developers: Because the Java GC moves objects around in the heap, RDMA in Java almost exclusively uses Off-Heap Memory (e.g., ByteBuffer.allocateDirect()). If the GC moved a memory block while an RDMA operation was in progress, the NIC would read/write to the wrong location, causing data corruption.

In standard networking, you open a “Socket.” In RDMA, you create a Queue Pair (QP) — Send Queue (SQ)+ Receive Queue(RQ). This serves as the direct communication channel between two applications. There are 3 main types of queues:

• Send Queue (SQ): Where the application posts instructions to send data or read remote data.
• Receive Queue (RQ): Where the application posts buffers to catch incoming data.
• Completion Queue (CQ): Where the NIC leaves a “note” saying an operation is finished.

Here is the lifecycle of a single RDMA “Write” operation (sending data to another machine):

• Step A: The application creates a Work Request (WR) (a small struct describing the job: “Take 10MB from Address X and put it at Address Y on the remote machine”). It pushes this request onto the Send Queue.
• Step B: The application cannot touch the hardware directly to start the job. Instead, it writes to a special memory-mapped register on the NIC. This tells the NIC, “Check the Send Queue, there is new work.”
• Step C: The NIC (specifically the DMA engine on the NIC) wakes up
  1. It reads the Work Request from the Send Queue
  2. It uses the Physical Address (from Step 1) to reach directly into the system RAM and pull the data. The CPU is not involved here.
  3. It packetizes the data and sends it over the wire.
• Step D: The remote NIC receives the packets. Because the memory was pre-registered, the remote NIC knows exactly where in its local RAM to place the data. It writes the data directly to memory. The remote CPU is not interrupted.
• Step E (Completion): Once the transfer is confirmed, the local NIC writes a Completion Queue Entry (CQE) into the Completion Queue. The application checks (polls) this queue to see that the send was successful.

RDMA supports two distinct modes of operation, which is very different from TCP/IP.

Two-Sided (Send/Receive)
This is similar to standard networking.
* Sender: Says “Send this.”
* Receiver: Must actively say “I am ready to receive this” by posting a buffer to their Receive Queue.
* Use Case: RPC calls or messaging where the receiver needs to be notified immediately.

One-Sided (Read/Write)
This is the true power of RDMA.
* RDMA Write: The sender writes directly into the receiver’s memory. The receiver doesn’t even know it happened (unless they check the memory). The receiver’s CPU is completely bypassed
* RDMA Read: The sender reaches into the receiver’s memory and pulls data back.
* Use Case: Ideally suited for accessing large shared datasets or distributed caches where you don’t want to disturb the remote server’s CPU just to read a value.

While most of us might find the Two-Sided model more familiar, for raw performance tuning, the One-sided model is often where the extreme low-latency gains are found, as it completely decouples the transfer from the remote application’s processing loop.

InfiniBand

InfiniBand (IB) is a high-speed networking communications standard designed specifically for the rigorous demands of High-Performance Computing (HPC) and modern AI data centers. It delivers far higher bandwidth and lower latency than Ethernet, and it natively supports RDMA, which is why it dominates HPC and large‑scale AI training environments.

While Ethernet was designed for compatibility and general-purpose use (e.g., connecting laptops, printers, and servers), InfiniBand is designed for one thing: speed. It provides the highest throughput and lowest latency possible between compute nodes

• It’s Lossless: InfiniBand networks are designed never to drop a packet of data due to congestion.
• It’s Specialized: You cannot plug an InfiniBand cable into a standard office network switch. It requires dedicated InfiniBand switches and specialized cabling.
• The Use Case: Pure supercomputing clusters, massive AI training rigs (like NVIDIA DGX SuperPODs), where performance is the only metric that matters.

Unlike Ethernet, which was built for standard packet switching and later adapted for RDMA (via RoCE), InfiniBand was built from the ground up with RDMA (Remote Direct Memory Access) as a core feature. This makes it the “gold standard” for low-latency communication.

InfiniBand uses a “switched fabric” topology. In a standard network, devices share a medium or connect via a hierarchy. In a fabric:
* Every node has a direct, dedicated path to other nodes via switches. Nodes connect to switches, which use internal high-speed paths (the “fabric”) to route data efficiently between ports based on destination addresses.
* It is designed to scale out easily (adding more switches and nodes) without degrading performance. Supports a vast number of devices by adding more switches or creating mesh topologies.
* Unlike shared broadcast networks, switched fabrics allow simultaneous, parallel communications between different pairs of nodes.

InfiniBand uses a channel‑based architecture with point‑to‑point serial links and switches. Each node connects via an HCA (Host Channel Adapter), which handles RDMA and messaging in hardware

Infiniband is lossless. This is a major differentiator from standard Ethernet: if the network becomes congested, switches can drop packets, and the sender has to wait for a timeout and resend the data. This destroys performance in HPC. Infiniband uses a “credit” system instead, where a sender cannot send a packet unless the receiver (specifically the switch buffer) explicitly confirms it has space (credits) to receive it. This creates a “lossless” network where packet drops due to congestion are mathematically impossible.

InfiniBand also pushes almost all network management tasks (routing, transport layer processing) to the hardware (the Switch and the Host Channel Adapter/HCA card). This frees up the server’s CPU to focus entirely on running your application logic.

Hardware components
An InfiniBand network looks different physically than a standard Ethernet setup:

• HCA (Host Channel Adapter): The InfiniBand equivalent of a Network Interface Card (NIC). These plug into the server's PCIe slot.
• Switches: Specialized hardware that routes InfiniBand traffic. They have extremely low “port-to-port latency” (often under 100 nanoseconds).
• Cables: InfiniBand uses high-grade copper (for short distances) or optical fiber (for long distances) with specific connectors (like QSFP).

InfiniBand speeds are named according to their data rate multiplier. This roadmap is aggressive and typically leads the industry ahead of Ethernet.

Mellanox (now NVIDIA Networking) is the primary vendor of InfiniBand HCAs and switches.

If you are building a distributed system using Java that requires massive throughput (like high-frequency trading or real-time analytics), InfiniBand is the hardware layer that removes the network bottleneck, allowing your software to run as if all the machines were sharing the same RAM stick.

RDMA over Converged Ethernet (RoCE)

RoCE (RDMA over Converged Ethernet) is a technology that brings ultra‑low‑latency, high‑throughput RDMA communication to standard Ethernet networks, letting servers exchange data directly between their memory with minimal CPU involvement. It delivers InfiniBand‑class performance while using familiar Ethernet infrastructure, which is why it’s widely used in modern data centers, AI clusters, and high‑performance storage systems.

Get Kaustubh Saha’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Essentially, RoCE tries to offer the “best of both worlds”: the extreme performance and low latency of InfiniBand combined with the cost-effectiveness and ubiquity of standard Ethernet infrastructure. Before RoCE, if you wanted RDMA speed, you had to buy InfiniBand switches, cables, and specialized administrator training.

InfiniBand is amazing, but building a separate, specialized network just for your servers is expensive and complex. Everyone already knows how to manage Ethernet — the standard networking technology used everywhere. RoCE was explicitly designed to bring InfiniBand‑style RDMA to Ethernet environments. It enables Remote Direct Memory Access (RDMA) over Ethernet, so it guarantees:

• Data movement directly between application memory buffers across machines
• No kernel involvement → no system calls, no context switches
• Zero‑copy transfers
• Very low CPU usage
• Microsecond‑level latency

The Challenge: InfiniBand is a “lossless” network (it never drops packets). Ethernet is a “lossy” network (it drops packets when congested). RoCE bridges this gap.

• How it works: RoCE takes those super-fast InfiniBand data packets and wraps them inside standard Ethernet packets. This tricks standard Ethernet switches into carrying high-speed, low-latency data.
• The Benefit: You get near-InfiniBand performance, but you can often use existing, cheaper Ethernet switches and cables (provided they are high quality).

This is ideal for workloads like distributed databases, AI training, and financial systems.

Versions of RoCE

There are two distinct versions of the protocol. It is important to distinguish between them as they operate differently on the network

RoCE v1 (The Layer 2 Protocol)

• It encapsulates the RDMA packet directly inside an Ethernet frame.
• It lacks an IP header. This means it is a Layer 2 protocol only. It cannot be routed.
• Works only within the same broadcast domain (same VLAN). You can only communicate with machines on the same physical subnet (same broadcast domain). If you have a router between two servers, RoCE v1 will not work.

RoCE v2 (The Routable Standard)

• It encapsulates the RDMA packet inside UDP/IP.
• Layer‑3 protocol using UDP/IPv4 or UDP/IPv6
• Because it has an IP header, it can be routed across Layer 3 networks. It can travel across the entire data center or even the internet (though latency would suffer).
• Uses UDP port 4791
• Adds congestion control using ECN (Explicit Congestion Notification)+ CNP (Congestion Notification Packet). ECN offers a proactive approach to congestion control. Instead of waiting for packet drops to indicate congestion (as traditional TCP does), ECN allows switches to mark packets when congestion is detected. Upon seeing these marks, the receiving endpoint sends a CNP back to the sender to reduce its transmission rate and throttle the flow.
• This is the modern industry standard for Ethernet-based RDMA.

Mellanox Cards: The Universal Translator

When people in the data center talk about “Mellanox cards” (Mellanox was acquired by NVIDIA in 2020, so they are officially NVIDIA ConnectX cards now), they are talking about the physical hardware — the Network Interface Card (NIC) — that you plug into the back of the server. Although NVIDIA acquired Mellanox in 2020, engineers still universally refer to them as “Mellanox cards.”

Mellanox cards (now officially NVIDIA Networking adapters) are specialized, high-performance hardware components that connect servers to networks. Unlike standard network cards, they are designed specifically to handle massive amounts of data with extremely low delay (latency), making them the industry standard for High-Performance Computing (HPC) and AI.

A standard network card (NIC) simply passes data to your CPU, which then processes it. A Mellanox card is a SmartNIC; it has its own onboard processor that handles network tasks, freeing your main CPU to run your application. Uniquely, most Mellanox cards feature Virtual Protocol Interconnect (VPI). Virtual Protocol Interconnect (VPI) is Mellanox/NVIDIA’s hardware architecture that allows a single network adapter port to operate as either InfiniBand or Ethernet, dynamically and without requiring hardware changes. It provides a unified NIC/HCA that supports both protocols, allowing data‑center operators to choose the best transport per port, per workload, or per cluster topology. This means a single card can be configured to act as an Ethernet or an InfiniBand card, depending on which cable/switch you plug it into.

Each port on a Mellanox VPI adapter can be configured as:
1. InfiniBand (IB) or,
2. Ethernet (including RoCE/RoCEv2) or,
3. one of each simultaneously (on dual‑port cards)

Before VPI, NICs were “single‑protocol”:
* IB cards couldn’t join Ethernet networks
* Ethernet NICs couldn’t join IB fabrics
This forced data centers into rigid, protocol‑specific deployments. VPI removes that rigidity by letting each port choose the optimal protocol for its segment of the network

Mellanox cards also offer a specific software advantage called VMA (Messaging Accelerator). VMA, technically known as libvma, is a dynamically linked library originally developed by Mellanox (now NVIDIA) that intercepts standard Java socket calls. It’s primary purpose is to dramatically reduce latency and increase throughput for standard socket-based applications (like Java apps) without requiring you to rewrite your code. Instead of sending the data through the slow OS Kernel (Context Switch), VMA routes the data directly to the Mellanox hardware (Kernel Bypass).
Result: You can potentially speed up standard multicast or UDP Java applications by 30–50% without changing a single line of code

Normally, when your Java app sends data: Java → JVM → Glibc (Socket) → Linux Kernel (TCP/IP Stack) →Driver → NIC.

With VMA, you use a Linux environment variable called LD_PRELOAD to load libvma.so before the standard system libraries. Java → JVM → libvma → User Space Driver (Verbs) →NIC. The Kernel, with its context switches and interrupts, is removed from the path.

Before installing VMA, you must have the NVIDIA/Mellanox OFED (MOFED) drivers installed. VMA relies on the InfiniBand Verbs interface provided by these drivers.

You can install VMA via the package manager if you have the Mellanox repositories configured

sudo yum install libvma libvma-utils

You do not configure VMA inside your Java code. You configure it in the environment where the Java process runs.

To run your Java application with VMA, you simply prepend the library path to your execution command.

# Standard run
java -jar my-trading-app.jar

# VMA run
LD_PRELOAD=libvma.so java -jar my-trading-app.jar

Note: On 64-bit systems, libvma.so usually lives in /usr/lib64/. If the command above fails, specify the full path: LD_PRELOAD=/usr/lib64/libvma.so ...

VMA is highly configurable via environment variables. You usually pass these in the same command line or export them in your .bashrc.

Common Settings for Low Latency:

• VMA_SPEC=latency: This is a preset profile. It optimizes the library for lowest possible latency (polling instead of waiting for interrupts).
• VMA_RING_ALLOCATION_LOGIC_TX=20: Optimizes memory allocation for transmission rings.
• VMA_THREAD_MODE=3: Crucial for multi-threaded Java apps. It controls how VMA handles locks. Mode 3 (Single Spin Lock) is often best for latency if you have specific thread pinning, but Mode 0 (Multi-Mutex) is safer for general multi-threaded apps.

VMA_SPEC=latency \
VMA_MTU=4096 \
LD_PRELOAD=libvma.so \
java -Djava.net.preferIPv4Stack=true -jar my-trading-app.jar

How do you know it’s working?

When the application starts, VMA prints a banner to stdout or stderr. If you see a logo like this in your Java application logs, VMA is active:

VMA_VERSION: 9.8.1-1
VMA_INFO: ---------------------------------------------------------------------------
VMA_INFO: VMA_SPEC=latency
VMA_INFO: ---------------------------------------------------------------------------

Using vma_stats NVIDIA provides a utility to monitor VMA performance in real-time

# Get the PID of your Java process
pidof java

# Watch VMA statistics
vma_stats -p <PID>

If this tool connects and shows packet counters increasing, your Java app is successfully bypassing the kernel.

Critical Considerations for Java Developers
While VMA is powerful, it introduces unique behaviors to the JVM:
1. CPU Usage (Polling) In “Latency” mode, VMA does not put the thread to sleep when waiting for a packet. It spins (polls) the CPU 100% of the time.

• Impact: Your Java threads doing network I/O will show 100% CPU usage in top. This is expected.
• Fix: Ensure you have enough CPU cores. Do not oversubscribe cores.

2. Thread Pinning (Affinity) Because of the 100% spinning, you must ensure your OS scheduler doesn’t move the Java thread to a different core.

• Use taskset or numactl to pin the Java process to specific cores (ideally on the same NUMA node as the Mellanox card).

3. Unsupported Java Features VMA generally supports standard java.net.Socket and java.nio.channels.SocketChannel. However:

• Selector.select(): VMA has its own implementation of select()/poll(). It usually works transparently, but complex NIO frameworks (like Netty) sometimes require specific VMA configuration flags to behave correctly.
• Use Case: VMA is best for UDP Multicast (Market Data) and UDP Unicast. While it supports TCP, the performance gains are sometimes less dramatic compared to the complexity of tuning it.

The Takeaway

When building high-performance infrastructure, you need low latency, which means you need RDMA.

You have two roads to get there: the dedicated, specialized track of InfiniBand, or the converged, flexible highway of RoCE.

Mellanox (NVIDIA) cards are the vehicle that lets you drive on either road, depending on your budget, your existing infrastructure, and just how fast you really need to go.