Ever Wonder Why RDMA Doesn’t Make Your Database Faster?
You have a distributed transaction that takes 60 milliseconds to commit. You profile it. Most of the time is “network.” So you do the obvious thing — you get NICs that can do a round trip in under two microseconds, you rip out the kernel networking stack, you wire up RDMA verbs, and you run the benchmark again.
It takes 59.9 milliseconds.
I hit exactly this wall while building the RDMA transport in mako-rdma, and the lesson turned out to be less about RDMA than about where in your commit path the latency actually lives. Everything works — the NIC does what it promises, the microbenchmarks are gorgeous — until you point it at a real transaction protocol and discover you optimized a rounding error.
Here’s the honest version of that story, including the parts where the fast network genuinely does win.
The number that doesn’t move
The reason kernel bypass didn’t help is not subtle once you write it down. A geo-replicated transaction that runs two-phase commit and waits for replication before it acknowledges has a wide-area round trip inside the critical path. Everything else is noise.
Swapping kernel TCP for a one-sided RDMA read saves you about 48.5 microseconds. Against a 50-millisecond WAN round trip, that is 0.097% of the commit path. You could make the datacenter network infinitely fast — latency zero, bandwidth infinite — and the transaction would still take 50 milliseconds.
The Mako authors say this plainly: RDMA’s benefits “are negligible compared to the long latencies that are fundamental to wide-area networking.” That sentence cost me a weekend to believe.
This is the same shape of mistake I wrote about in Why Scaling Backend Systems Is Harder Than You Think — you speed up the thing you can measure instead of the thing that’s serialized.
What people try first, and why it fails
The instinct after the transport swap fails is to go further down the same road: eliminate the remote CPU entirely with one-sided RDMA. The client reads the server’s memory directly. No remote thread wakes up, no request is parsed, no CPU cycle is spent on the far side. On paper this is the strongest form of the idea.
Two things break it.
First, one-sided reads can’t follow pointers. Your index is a tree. A one-sided read fetches bytes at an address you already know — so every level of descent is another network round trip, because you can’t know the child pointer until the parent comes back. In mako-rdma the storage layer is Masstree; a lookup that costs one two-sided RPC becomes three or four one-sided round trips.
// One-sided descent: each level is a separate RTT.
// The remote CPU is idle, and that is exactly the problem.
uint64_t addr = root_addr;
for (int level = 0; level < kTreeHeight; ++level) {
struct ibv_sge sge = {
.addr = reinterpret_cast<uintptr_t>(node_buf),
.length = kNodeSize,
.lkey = local_mr->lkey,
};
struct ibv_send_wr wr = {};
wr.wr_id = level;
wr.sg_list = &sge;
wr.num_sge = 1;
wr.opcode = IBV_WR_RDMA_READ;
wr.send_flags = IBV_SEND_SIGNALED;
wr.wr.rdma.remote_addr = addr;
wr.wr.rdma.rkey = peer.rkey;
struct ibv_send_wr *bad = nullptr;
if (ibv_post_send(qp, &wr, &bad)) return -errno;
struct ibv_wc wc;
while (ibv_poll_cq(cq, 1, &wc) == 0) {} // spin: ~1.5 µs
if (wc.status != IBV_WC_SUCCESS) return -EIO;
addr = child_of(node_buf, key); // pointer only now known
}Four iterations of that loop is ~6 µs of pure round trips, versus ~2.3 µs for a single two-sided RPC where the remote core does all four hops in local DRAM. The “free” version is slower.
Second, and worse, reliable connections don’t scale. RDMA’s reliable-connected transport keeps per-connection state in NIC SRAM. The eRPC authors measured it: roughly 375 bytes of NIC SRAM per connection, against a NIC budget of about 2 MB. At 5,000 connections you need ~1.8 MB and the cache starts missing — and every miss is a DMA read across PCIe to fetch connection state before the packet can move.
Connection-state and throughput figures from Kalia et al., NSDI ‘19.
The part I find genuinely unsettling: they note connection scalability on ConnectX-5 was “surprisingly, not substantially better” than on Connect-IB from five years earlier. This is not a driver bug waiting to be fixed. It’s on-die SRAM, and SRAM is expensive.
Get Goutham Kalla’s stories in your inbox
Join Medium for free to get updates from this writer.
Now count your connections. A sharded transactional store is all-to-all by construction: shards × cores × peers. Ten shards, 32 cores each, full mesh — you are in the thrashing region before you have a large cluster.
What actually works: take the WAN out of the commit path
The fix isn’t a faster wire. It’s refusing to put the wide-area round trip in the critical path at all.
This is what Mako does, and it’s the reason the RDMA transport work becomes worth doing afterward rather than instead. Mako decouples transaction execution from replication: shard leaders run 2PC certification among themselves and, once certified, make writes speculatively visible to subsequent transactions. Replication to remote regions proceeds asynchronously on per-core Paxos streams. The client isn’t waiting on Oregon.
The reported result, from the OSDI ’25 paper: 3.66M TPC-C transactions/sec at 10 shards, against Calvin at 0.42M — 8.6× — with median latency of 121 ms at batch size 600, of which 50 ms is the WAN RTT and 13 ms is batching.
All figures reported by Shen et al., Mako, OSDI ’25. These are the paper’s measurements on Azure VMs with 32 vCPUs and 16 Gbps accelerated networking — not mine.
Here is the turn that makes the whole exercise worthwhile. Once the WAN is off the critical path, the bottleneck moves back into the datacenter — leader-to-leader 2PC messages, Paxos stream fan-out, packet processing cost per core. Those are microsecond-scale paths now, and at 3.66M transactions per second you are issuing tens of millions of messages per second. Suddenly 50 µs of kernel TCP per message is not a rounding error; it’s the whole budget.
That’s why Mako runs DPDK and eRPC intra-datacenter, and it’s the correct order of operations: protocol first, transport second. Kernel bypass is a throughput optimization masquerading as a latency optimization. It pays when your cores are burning cycles on packets, not when your packets are burning milliseconds on physics.
The two-sided version of that read, in eRPC, is also just shorter:
rpc.enqueue_request(session_num, kReqTypeGet,
&req_msgbuf, &resp_msgbuf,
get_continuation, /*tag=*/req_id);
rpc.run_event_loop_once(); // remote core walks the tree locallyThe ugly part nobody puts in the abstract
Speculation is not free, and this is where I’d push back on anyone who reads the throughput number and stops.
If a transaction reads a value written by a speculative transaction that hasn’t replicated yet, and that speculative transaction later fails, everything downstream must abort. Mako doesn’t prevent cascading aborts — it bounds them, which is a meaningfully weaker and more honest claim. Each transaction carries a version vector clock with one entry per shard, maintaining the invariant that if T1 depends on T2, T1’s vector clock dominates T2’s pairwise. Followers track a vectorized watermark of replication progress, and a transaction can only be replayed once its version falls below that watermark.
Failure recovery works in epochs. A replicated configuration manager detects a dead leader by heartbeat, elects a replacement, and broadcasts an epoch increment. Recovering leaders pull replicated log entries and commit no-ops for entries they can’t recover; healthy shards append an INF marker. A Finalized Vector Watermark defines a consistent global cut, and everything above it is rolled back. Healthy shards keep serving new-epoch transactions rather than freezing — they just can’t read from uncertain old-epoch state.
That’s a well-designed protocol, and it still means a leader crash in one region can retroactively abort committed-looking work in another. If your application has already emailed the customer, “speculatively visible” is a load-bearing adjective. That constraint belongs in your API docs, not a footnote.
The paper’s own limits are worth repeating: static sharding only, no dynamic resharding; no follower reads; vector clocks become a bottleneck past roughly 320 shards, and the compression that relieves it degrades fault tolerance; one-shot transactions only, no interactive multi-round ones.
What I’d do differently
I built the transport layer before I understood the protocol’s latency structure, which is the wrong order — and it’s the same instinct that makes people reach for Byzantine agreement when they haven’t yet established what failure model they’re actually in, something I got into in N Generals Problem. I’d now start by drawing the commit path and marking which segments are serialized against a client-visible acknowledgment. Anything not on that line is a throughput question, and throughput questions have different answers than latency questions.
The thing I still can’t resolve: the vector clock is O(shards) per transaction, so the metadata grows with the cluster while the payload doesn’t. Past a few hundred shards you’re shipping more dependency-tracking bytes than data. Compression trades fault tolerance for it. Coarser granularity trades false dependencies for it.
So here’s the open question I’d actually like an answer to: is there a dependency-tracking representation whose size grows with a transaction’s actual dependency footprint rather than with the cluster’s shard count — and can it be computed without a round trip to find out what that footprint is? Every scheme I’ve sketched needs to know the footprint before it can be small, and needs to be small before it can avoid the round trip that would tell it.
check out my repository for implementation :
https://github.com/kallagoutham/mako-rdma
Stay tuned, Happy vibecoding!!
