Passport Control: How RoCEv2 Packets Learned to Cross Borders

Story so far —

In my last post, I showed you what happens once a packet lands on the NIC and makes its way into GPU memory. Fair question you might’ve had while reading that: how does the NIC even know which packets are RDMA packets? Time to rewind a bit and open the packet up.

Los geht’s!

1. RoCE vs RoCEv2 — a short history-lesson detour—

InfiniBand originally didn’t care about Ethernet or IP at all; it had its own entire world. RoCE (the first version) was the compromise: take InfiniBand’s transport layer and run it over Ethernet, instead of native IB switches. RoCEv2 pushes that compromise one step further, changing how addressing works so RDMA traffic becomes routable across normal IP networks too, not just a single Ethernet segment.

Pic Credits — FS.com

The Ethertype field is what tells a receiver which of the two worlds it’s looking at, RoCE or RoCEv2, right at the start of the packet. How that addressing swap actually works, and why it’s the one change that matters most, is worth cracking a packet open properly for.

So the whole point of RoCEv2 in one line: same InfiniBand transport guarantees, but now it’s routable, because it borrowed IP’s passport.

Time for a short break. It’s getting hotter in Germany these days 🥵

2. Cracking open a single packet —

Source — Nvidia RoCEv2 Considerations

Before talking about the main guy, Base Transport Header (BTH), though, worth finishing the addressing story from Part 1, because it explains why BTH ends up looking the way it does.

Native InfiniBand actually used two separate routing headers, depending on how far a packet needed to travel. LRH (Local Route Header) handled hops within a single subnet, GRH (Global Route Header) handled hops across subnets. RoCE(v1) kept this exact split, it just swapped LRH out for a plain Ethernet L2 Header(since there’s no native IB switch fabric anymore), while keeping GRH as-is for anything crossing a subnet boundary.

RoCEv2 is where the second half finally changes too. GRH gets dropped entirely, replaced by a plain IP Header + UDP Header. And there’s one specific number doing a lot of quiet work here: the UDP destination port is fixed at 4791. That’s the entire trick, as far as any ordinary router or switch in between is concerned, this is just a UDP packet headed to port 4791, nothing exotic, route it like anything else on the network. It’s only once the packet reaches a RoCEv2-aware NIC that port 4791 gets recognized for what it really is, a signal that says, “don’t treat this as UDP payload, what follows is an InfiniBand BTH.”

NIC finally recognises his friends, and RoCEv2 packets and NIC have a happy ending, always!

So, quick mental model: LRH is always replaced by Eth L2 Header. GRH is kept as-is in RoCEv1, replaced by IP + UDP (port 4791) in RoCEv2. That second swap is the main reason why your RoCEv2 packets can be routed across datacenter IP Infrastructure.

From here on, everything remains the same whether you are in the native IB world or in the new world of RoCEv2.

Every InfiniBand packet, RoCE or not, carries a Base Transport Header (BTH). This is the one header that’s always there, and it’s carrying the essentials —

• Opcode — what kind of operation is this (RDMA_WRITE, RDMA_READ, SEND, RECV)
• Destination QP -- which queue pair this packet belongs to
• PSN(Packet sequence number) — Keeps ordering and reliable delivery honest
• Acknowledge request flag — does the receiver need to ACK this packet specifically?
• FECN/BECN — congestion notification bits, so both ends know when the network’s getting cranky

3. What happens when your payload is bigger than one packet —

Get Abhishek Mishra’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

InfiniBand MTU sizes only come in four flavours: 1024, 2048, 3072 or 4096 bytes. So step one, always, is comparing your payload size against the chosen MTU.

• If it fits in exactly one packet, the whole thing goes out with opcode SEND_ONLY. Simple, one packet, done.
• If it doesn’t, the payload gets carved up, and now the opcode tells a little story across multiple packets: the first one is SEND_FIRST, everything in the middle is SEND_MIDDLE, and the very last chunk is SEND_LAST.

The receiving side effectively reconstructs your payload just by watching that sequence of opcodes go by, no separate reassembly header required.

4. Before any of this can happen, your code has to build a small skyscraper —

Here’s the part that felt like a lot of ceremony at first, and now feels obvious in hindsight: you can’t just “send a packet”. You have to construct a hierarchy of resources first, and libibverbs is very particular about the order.

The chain looks roughly like this —

And in code, the sequence goes —

1. ibv_open_device (ib_dev) — grab the context for a specific NIC port
2. ibv_alloc_pd(context) — create a protection domain, think of it as a security boundary, everything registered under one PD can talk to everything else in that same PD, nothing outside it
3. ibv_reg_mr(pd, buffer, size, access_flags) — register the actual memory. In our case, this wasn’t the host RAM, it was GPU device memory, allocated with CU_POINTER_ATTRIBUTE_SYNC_MEMOPS set on it so the NIC could safely ttreat it as a valid peer-to-peer RDMA target.
4. ibv_create_cq(context, cqe, cq_context, channel, comp_vector) — set up the completion queue that will report back when work actually finishes
5. ibv_create_qp(pd,init_attr) — finally, the queue pair itself, the actual “endpoint” your application talks to, roughly the RDMA equivalent of a socket

Once the QP exists, you build an SGE list against it(address, length, lkey per buffer), pull the remote side’s GID(an IPv4-mapped IPv6 address, RoCEv2’s way of doing addressing), and only then start moving the QP through its state machine.

5. The QP state machine, briefly -

A freshly created QP can’t do anything yet. It has to move through states —

• Init — the QP exists but isn’t ready for traffic
• RTR(Ready to Receive) — now it can accept incoming data. This is also the point where you’d post your receive work requests, so the hardware has somewhere to actually put incoming payloads the moment they arrive
• RTS(Ready to Send) — only needed on the sending side, and only after RTR, because you want your receive path laid down before you start firing packets at someone

After that, it’s mostly a loop: post receive WRs ahead of time, poll the completion queue(ibv_poll_cq) continuously, and every CQE that shows up tells you one more piece of data has landed successfully.

So next time a packet lands cleanly in GPU memory like I showed in the last post, this is the machinery that had to exist before the NIC even saw the first byte, a protection domain, a pinned registered memory, a completion queue, and a queue pair that already knew where incoming data should go.

Next up, I’ll mostly zoom out from single packets and walk through the actual RX/TX streaming architecture I built around all this, the piece that ties everything together end to end. Latency hunting comes right after that, once there’s a full pipeline to point a profiler at.

See ya super sooooon!!