RDMA — The Secret Network Behind AI Supercomputers [Article 2]
RDMA - The Secret Network Behind AI Supercomputers
✅ Article 1: Why Ethernet Starts to Struggle at Scale
🟡 Article 2: How a NIC Can Talk to Memory
⬜ Article 3: The Journey of a Tensor — Following Every Byte from GPU Memory to the Network
⬜ Article 4: RDMA From Scratch — Speaking the Language of the NIC
⬜ Article 5: RoCEv2 and InfiniBand — The Networks Powering AI SupercomputersHow a NIC Can Talk to Memory
In the first article, NovaAI discovered that traditional Ethernet was not really the problem.
The real problem was too much work around the data.
The CPU was spending time copying bytes, managing buffers, handling the kernel networking stack, and coordinating transfers that should have been much cheaper.
So NovaAI asked a bold question:
Can the NIC move data more directly, without making the CPU do so much work?
That question is the beginning of RDMA.
But before we can understand RDMA, we need to understand something more basic:
How can a network card talk to memory at all?
That sounds simple.
It is not.
Because memory is not just a pile of bytes sitting in a box.
Modern computers protect memory, translate addresses, and keep devices under control. If a NIC is going to access memory directly, the system must decide exactly which memory, how, and under what rules.
This article explains that step by step.
The NovaAI problem, in plain words
One Server
│
│
+-------------------------------+
| |
| Application |
| │ |
| ▼ |
| Memory |
| ▲ |
| │ |
| CPU |
| ▲ |
| │ |
| PCIe |
| ▲ |
| │ |
| NIC |
+-------------------------------+
│
Ethernet Network
│
+-------------------------------+
| Another Server |
+-------------------------------+NovaAI has many servers. Some of them run training jobs. Some move data. Some talk to storage. Some exchange results with other machines.
At first, everything worked well enough over normal TCP/IP networking.
But as the cluster grew, the same pattern kept showing up:
- the CPUs were busy doing networking work
- the GPUs were waiting for data
- the network was moving packets, but not efficiently enough
- every data transfer seemed to involve too many steps
NovaAI did not want the CPU to act like a courier for every transfer.
It wanted the NIC to do more of the real work.
To understand how that is possible, we need to understand how memory access works inside a modern machine.
Memory is not as simple as it looks
Application thinks
+--------------------------------+
| 0x1000 |
| 0x1001 |
| 0x1002 |
| 0x1003 |
+--------------------------------+
Reality inside RAM
+---------+ +---------+ +---------+
| Page 14 | | Page 92 | | Page 41 |
+---------+ +---------+ +---------+
(Not physically next to each other)When an application asks for memory, it usually does not get a raw physical address.
It gets a virtual address.
That is an important idea.
A virtual address is the address the application sees. It is the address used in code, pointers, and buffers. But it is not always the actual location in RAM.
The operating system and hardware work together to translate that virtual address into a real memory location.
That gives us:
- safety
- isolation
- flexibility
- easier programming
So a program might think it owns a neat block of memory, even though the real RAM may be arranged differently underneath.
Here is the basic idea:
Application sees:
0x1000, 0x1001, 0x1002, ...But the real RAM may be:
page A here
page B somewhere else
page C laterThe application does not usually care.
The CPU and operating system care very much.
The CPU uses the MMU to translate addresses
The MMU is the Memory Management Unit.
You do not need to remember the acronym at first. Just remember the job:
The MMU helps the CPU turn virtual addresses into physical addresses.
So when the CPU wants to read or write memory, it does not blindly guess where the data is. It asks the MMU to translate the address safely.
A simple picture:
Application
Virtual Address
│
▼
+--------------+
| MMU |
+--------------+
│
▼
Physical Address
│
▼
RAMThis is how the CPU knows where data really lives.
But here is the problem:
A NIC is not the CPU.
So the NIC cannot automatically use the same translation system in the same way the CPU does.
That leads us to the next question.
Why a NIC cannot just touch any memory it wants
Wrong ❌
NIC
│
│
▼
Entire RAM
+----------------------------+
| OS Memory |
| Chrome |
| Database |
| AI Model |
| Passwords |
| Everything |
+----------------------------+Obviously this would be a disaster.
Imagine if a NIC could write to any memory address it liked.
That would be dangerous.
It could overwrite:
- another application’s buffer
- the operating system
- security-sensitive memory
- data belonging to another process
That would be chaos.
So the system must control device access very carefully.
The NIC needs permission before it can touch application memory.
That permission is one of the most important ideas behind RDMA.
The NIC connects through PCIe
Before a NIC can do anything useful, it needs a physical path into the machine.
That path is usually PCIe.
PCIe is the high-speed internal bus that connects devices like:
- network cards
- storage controllers
- GPUs
- accelerators
A simple mental model:
CPU
│
│
Memory Controller
│
+-----------+------------+
| |
▼ ▼
RAM PCIe Root
│
│
+------------+------------+
| |
▼ ▼
NIC GPUThe NIC sits on the PCIe bus and communicates with the host system through it.
But PCIe is just the road.
It does not itself explain how the NIC gets permission to use memory.
That is where DMA comes in.
DMA: the NIC moves data for us
DMA means Direct Memory Access.
This is the idea that a device can move data directly to or from system memory without the CPU copying every byte by hand.
That is a huge deal.
Without DMA, the CPU has to do more of the movement itself.
With DMA, the CPU can prepare the transfer, then let the hardware carry it out.
A simple comparison:
Without DMA:
Application
│
▼
CPU copies data
│
▼
RAM
│
▼
NICWith DMA:CPU
│
│ Configure DMA
▼ RAM
▲
│
│ PCIe DMA Transfer
▼NIC
This does not mean the CPU disappears.
It means the CPU stops doing the boring part of the work.
That matters a lot at scale.
NovaAI wants the CPU to spend time on real application logic, not on repeatedly copying data around.
How the NIC actually touches memory
Now comes the big question:
How does the NIC know which memory it is allowed to use?
The answer is: it does not get free access to everything.
The memory has to be prepared first.
This preparation is called memory registration.
CPU
│
Register Memory Region
│
▼
+-----------------------+
| System RAM |
| |
| Registered Buffer |
+-----------+-----------+
▲
│
DMA Read / Write
through PCIe
│
▼
+----------------+
| NIC |
| DMA Engine |
+----------------+Memory registration: telling the system what the NIC may use
Application
Allocate Buffer
│
▼
+--------------------+
| Application Buffer |
+--------------------+
│
▼
Register Buffer
│
▼
NIC now knows
✔ Start Address
✔ Size
✔ PermissionsMemory registration is like saying:
“This buffer will be used for direct high-speed access. Please make it ready for the NIC.”
That is all.
But that one step is extremely important.
Why?
Because the NIC needs:
- a valid memory region
- stable addresses
- permission to access it
- protection so it cannot touch other memory by mistake
So when memory is registered, the system marks it as safe and usable for direct device access.
Get Gautam Vasudeva’s stories in your inbox
Join Medium for free to get updates from this writer.
At a simple level:
Application allocates buffer
|
v
Buffer is registered
|
v
System prepares it for device access
|
v
NIC can use it safelyThis is one of the biggest differences between normal socket networking and RDMA-style networking.
In ordinary networking, the kernel often manages the data path for you.
In RDMA, the memory is prepared more explicitly so the NIC can work with it directly.
Why memory is often pinned
When memory is registered, it is often pinned.
Pinned memory means the operating system will not casually move it around while the NIC is using it.
Normal Memory
Page A
↓
OS moves it
↓
Page B
Pinned Memory
Page A
↓
Cannot Move
↓
DMA always knows
where it isWhy does that matter?
Because the NIC needs the memory location to stay stable.
If the memory moved in the middle of a transfer, the NIC could lose track of it.
So the system says:
- this memory region is special
- keep it in place
- let the device use it safely
This improves performance and reliability, but it also means the system has less freedom to reshuffle memory behind the scenes.
That is a normal tradeoff in systems engineering:
more control for the hardware, less flexibility for the OS.
The IOMMU: protection for devices
The CPU has the MMU.
Devices often need the IOMMU.
The IOMMU is similar in spirit to the MMU, but for hardware devices.
Its job is to help control what a device can access.
That way, the NIC cannot just read or write random memory. It can only access memory the system has allowed.
This is important because direct hardware access is powerful.
And power must be controlled.
A simple picture:
NIC
│
▼
+-----------+
| IOMMU |
+-----------+
│
Allowed?
──────────────
YES
↓
Registered Buffer
NO
↓
Access BlockedThis helps keep the machine safe while still allowing fast transfers.
NovaAI needs this because it wants performance and security and stability.
It does not want one fast transfer to break the whole system.
Bus mastering: the NIC becomes active
Once the NIC is allowed to perform DMA, it can act like a bus master.
That means it can initiate transfers on the bus instead of waiting for everything to be copied by the CPU.
Think of it like this:
- the CPU says: “this buffer is ready”
- the NIC says: “good, I will move the data now”
- the data movement happens directly
A simplified path:
CPU prepares memory
|
v
Memory is registered and pinned
|
v
NIC performs DMA over PCIe
|
v
Data moves directlyThis is the first step toward RDMA.
But we still need to understand one more thing.
Memory is often not one perfect continuous block in the way people imagine it.
Real memory is often fragmented
In a real machine, memory can be scattered around.
The system may give an application a buffer that looks continuous from the outside, but internally it may be built from several pages.
That is normal.
To handle this, devices often use scatter-gather techniques.
The idea is simple:
- the buffer may be made of several pieces
- the device knows how to follow those pieces
- the transfer still works efficiently
This is useful because production systems are not always tidy.
Memory is often fragmented.
Applications grow and shrink.
Workloads change.
The system has to handle all of that without falling apart.
Put the whole picture together
Here is the simplest way to remember the entire memory access model:
- The application asks for memory.
- The application sees a virtual address.
- The CPU uses the MMU to translate virtual memory to physical memory.
- The memory region is registered so the NIC is allowed to use it.
- The memory is pinned so it stays stable.
- The IOMMU helps protect device access.
- The NIC uses DMA over PCIe to move data directly.
That is the foundation.
And now RDMA starts to make sense.
Because RDMA depends on this foundation.
Why all of this matters to NovaAI
NovaAI is not just moving small messages.
It is moving large amounts of data between machines all the time.
If the CPU has to copy every transfer, performance drops.
If the NIC can move data directly, the system becomes much more efficient.
That means:
- lower CPU usage
- lower latency
- better throughput consistency
- less wasted work
- more room for real application processing
This is why memory registration, DMA, PCIe, and device protection matter so much.
They are not side details.
They are the reason high-performance networking works at all.
A very simple diagram of the idea
Here is the easiest picture to keep in your head:
Application
│
Allocate Memory
│
Virtual Address
│
MMU
│
Physical Memory
│
Register Buffer
│
Pin Memory
│
IOMMU Protection
│
NIC DMA Engine
│
PCIe DMA Transfer
│
EthernetThat is the bridge between ordinary computing and RDMA-style networking.
What we still have not explained
At this point, we know how a NIC can safely talk to memory.
But we still do not know how the application tells the NIC what to do next.
How does it say:
- send this data
- receive that data
- write into that memory
- read from that memory
- notify me when you are done
That is where the real RDMA machinery begins.
We are now ready for the next layer:
- queue pairs
- work requests
- completion queues
- memory keys
- send and receive operations
- RDMA read and write
Those topics are the heart of RDMA.
And they deserve their own article.
Takeaway
The secret behind RDMA is not magic.
It is careful engineering.
The application uses virtual memory.
The CPU translates addresses with the MMU.
The system registers and pins the memory region.
The IOMMU protects device access.
The NIC uses DMA over PCIe to move data directly.
Once you understand that chain, RDMA stops feeling mysterious.
It becomes a logical next step.
And that is exactly where Article 3 will begin.
![RDMA — The Secret Network Behind AI Supercomputers [Article 1]](https://miro.medium.com/v2/resize:fit:1358/format:webp/7*hI1r_cRdF2SRJ9Wv_XVzYA.jpeg)
