Hands-On with RoCE: Simulation & Packet Analysis

With almost two decades in computer networking, I used to jokingly describe myself as network plumber 🧑‍🔧 and packet juggler 🤹. In recent years, however, my focus has shifted toward cloud-native solutions.

Wanting to reconnect with my infrastructure roots, I enrolled in NVIDIA’s AI Infrastructure and Operations Fundamentals course. One topic repeatedly stood out is RDMA (Remote Direct Memory Access), a key technology for high-performance communication between GPUs and other compute nodes in modern AI infrastructure.

While I’m highly experienced in datacenter and telco networks, RDMA was relatively unfamiliar to me. The course explained the concepts well, but it remained high-level. I wanted to see RDMA in action — capture its packets, dissect them, and understand how it behaves on the wire. So I set out to build a quick lab to explore it hands-on.

Get Charles Chan’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

There are two common deployment paths for RDMA:

• InfiniBand (IB) — A high-throughput, low-latency interconnect protocol with its own link, network, and transport layers. It requires specialized host channel adapters (HCAs) and switching hardware.
• RoCE (RDMA over Converged Ethernet) — An implementation of RDMA that runs over standard Ethernet networks, reducing the need for dedicated InfiniBand infrastructure. RoCE v1 operates at Layer 2. RoCE v2 (more common today) runs over UDP/IP, making it routable.

In this exercise, I will simulate RDMA using Soft-RoCE (rdma_rxe), capture the traffic, and analyze it in Wireshark.

Part 1: Lab Environment Setup

1. Create Two Azure VMs

• Ubuntu 24.04 LTS
• Size: B1ms (or similar)
• Place both VMs in the same virtual network and subnet to simplify routing.

2. Install Required Packages

We’ll use ibverbs-utils (userspace tools for RDMA verbs) and enable a generic kernel. Default Azure kernel omits some modules (like rdma_rxe) to reduce footprint for cloud workloads. The generic kernel includes them.

sudo apt update
sudo apt install ibverbs-utils linux-image-generic
sudo apt remove --purge linux-image-azure linux-image-6.11.0-1018-azure
sudo update-grub
sudo reboot

3. Enable Soft-RoCE

sudo modprobe rdma_rxe 
sudo rdma link add rxe0 type rxe netdev eth0

This creates a software RDMA device (rxe0) bound to the Ethernet interface (eth0).

4. Verify RDMA Device

Run ibv_devinfo to verify RDMA device. You should see an output like:

$ ibv_devinfo
hca_id: rxe0
        transport:                      InfiniBand (0)
        fw_ver:                         0.0.0
        node_guid:                      0222:48ff:feb6:cb0b
        sys_image_guid:                 0222:48ff:feb6:cb0b
        vendor_id:                      0xffffff
        vendor_part_id:                 0
        hw_ver:                         0x0
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

Part 2: Generate RDMA Test Traffic

We’ll use ib_write_bw, a bandwidth benchmarking tool installed with ibverbs-utils.

# On VM 1
ib_write_bw

# On VM 2
ib_write_bw <vm1 ip>

You should see output similar to the following, which confirms the RDMA connectivity between two VMs:

---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : rxe0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : OFF
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 1
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0014 PSN 0x6e8660 RKey 0x0005d9 VAddr 0x0072725d730000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:02:00:05
 remote address: LID 0000 QPN 0x0014 PSN 0x214b11 RKey 0x000540 VAddr 0x007c44e373f000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:02:00:04
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      5000             52.48              52.26              0.000836
---------------------------------------------------------------------------------------

Part 3: Capture & Analyze RDMA Traffic

1. Capture packets

Since we’re running Soft-RoCE v2, the RDMA packets are encapsulated over UDP (port 4791)

# On VM 1
sudo tcpdump -nei eth0 udp port 4791 and src <vm2_ip> -w roce.pcap

• Soft-RoCE enables RDMA testing without special NICs, perfect for learning environments.
• RoCE v2 traffic is UDP-encapsulated, making it routable across subnets (unlike RoCE v1).

2. Connect Request

Here are a few key observations from the initial handshake message, including differences between InfiniBand and RoCE:

• InfiniBand header is encapsulated inside a standard UDP header (port 4791).
• Only BTH (Base Transport Header) and ETH (Extended Transport Header) are present. LRH (Local Routing Header) and GRH (Global Routing Header) are absent, as their functions are replaced by TCP/IP.
• The CM Connect Request message in the handshake shows the VM’s IP address used as GID (Global Identifier), and a placeholder for LID (Local Identifier), since L2 and L3 addressing are handled by TCP/IP.

3. RDMA Write

Key observations from the RDMA Write message:

• Similar to the initial handshake — only BTH and ETH are present.
• Data length is 1024 bytes, matching the default MTU seen earlier in ibv_devinfo.

Bonus: How is This Different from InfiniBand?

What we built here with Soft-RoCE is a functional simulation of RDMA, but it’s not equivalent to running on real InfiniBand (IB) hardware.

Difference between InfiniBand and RoCE

Credentials