Why AI and HPC Need High-Speed Networking: Before Learning InfiniBand
AI Networking from the Wire to the GPU - Part 1
If you have been learning about AI infrastructure, you have probably heard some of these terms:
InfiniBand. RDMA. RoCE. ConnectX. NVLink. NVSwitch. GPUDirect. NCCL. BlueField. Quantum.
And if you are new to this space, they can quickly turn into alphabet soup.
You search for “What is InfiniBand?” and get told that it is a high-speed, low-latency interconnect used in HPC and AI.
Okay.
But why do we need it?
Ethernet already exists. TCP/IP runs practically the entire Internet. Modern Ethernet links can deliver hundreds of gigabits per second.
So why did the HPC world build something different? Why are networking technologies such a big deal in modern AI clusters? And why does NVIDIA, a company most people associate with GPUs, have an entire networking ecosystem around technologies such as ConnectX, Quantum, BlueField, and Spectrum?
Before we answer any of those questions, we need to understand the problem they are trying to solve.
And that is exactly where I want to start this series.
What I want to do with this series
I don’t want this to be a series where I throw definitions at you.
InfiniBand is this.
RDMA is that.
Queue Pair means this.
Run this command.
Congratulations, next topic.
You can find definitions anywhere.
My goal is to build the whole thing from the ground up.
We are eventually going to go quite deep: InfiniBand hardware, HCAs, switches, port states, Subnet Managers, LIDs and GIDs, RDMA, memory registration, Queue Pairs, verbs, packet structure, flow control, Virtual Lanes, routing, NVIDIA’s networking stack, GPUDirect RDMA, NCCL, performance testing, troubleshooting, and finally the design of actual AI/HPC fabrics.
But I want every new concept to answer a question created by the previous one.
By the end, when you see something like:
I don’t want it to look like a stack of mysterious technologies.
I want you to understand why every box exists and what the data is actually doing there.
So we’re going to start with none of them.
Let’s start with a computer.
One computer is easy
Imagine you have an application running on a server.
At a ridiculously simplified level, we could draw the system like this:
+------------------+
| Application |
+---------+--------+
|
v
+------------------+
| CPU |
+------------------+
|
v
+------------------+
| Memory |
+------------------+The application needs some computation. The CPU performs it. Data is stored in memory.
Of course, a real computer is vastly more complicated, but this model is enough for now.
Then our workloads get bigger.
We might add GPUs:
+------------------+
| Application |
+---------+--------+
|
+-----+-----+
| |
v v
CPU GPU
\ /
\ /
+-------+
|
MemoryNow we have significantly more compute capability. Great.
But what happens when the problem becomes too large for one server?
Maybe we need more memory.
Maybe one GPU isn’t enough.
Maybe eight GPUs aren’t enough.
Maybe finishing the workload on one machine would simply take too long.
This is where things become interesting.
We add another machine.
The moment you add another server, everything changes
Now our system looks something like this:
+----------------------+ +----------------------+
| Server A | | Server B |
| | | |
| CPU GPU | | CPU GPU |
| | | | | | | |
| +--Memory--+ | | +--Memory--+ |
+----------+-----------+ +----------+-----------+
| |
+------------- NETWORK -------------+We have doubled our compute resources. But we have also created a new problem.
Server A and Server B are physically separate computers.
Server A cannot simply reach into Server B’s memory as though it were its own.
If the two machines are working on different parts of the same problem, somehow information has to move between them.
And what sits between them?
The network.
This leads to one of the most important ideas I want you to remember throughout this series:
Once computation is distributed across machines, the network becomes part of the computer.
Not literally, of course. But from the workload’s perspective, it might as well be.
The material that motivated this series makes the same connection: distributed workloads run across multiple machines and therefore depend on fast node-to-node communication.
And modern AI takes this problem to another level.
Now imagine 1,000 GPUs
When we hear about an AI model being trained on thousands of GPUs, it’s tempting to imagine something like this:
Give every GPU some work.
Let them calculate independently.
Combine everything at the end.
If only it were that simple.
The GPUs frequently need to communicate.
Depending on how a workload is distributed, GPUs may need to exchange intermediate results, synchronize information, move tensors, aggregate gradients, distribute parameters, or coordinate different pieces of the model.
You don’t need to understand any of those mechanisms yet.
We’ll get there.
For now, imagine four workers solving one large problem.
Worker 1 Worker 2
\ /
\ /
\ /
COORDINATE
/ \
/ \
Worker 3 Worker 4If they could work completely independently for eight hours and exchange answers once at the end, communication wouldn’t be particularly difficult.
But suppose they have to exchange results every few seconds before continuing.
Suddenly, communication matters a lot.
That’s closer to the problem we’re dealing with.
Your GPU can be incredibly fast, but if its next computation depends on information coming from another GPU, it may have to wait for that information.
This frequent synchronization between GPUs and nodes is one of the major networking demands of AI/ML workloads.
This is also why I don’t particularly like thinking about AI infrastructure as simply:
Faster GPU = Faster AIThe real picture is closer to:
+-------------+
| Compute |
+------+------+
|
+-------------+-------------+
| | |
v v v
Memory Storage Network
\ | /
\ | /
+-----------+-----------+
|
v
Workload PerformanceEvery component feeds the others.
Imagine owning a sports car capable of 200 mph but driving it through traffic where the average speed is 25 mph.
The engine isn’t the problem.
The environment around the engine prevents you from using its capability.
A high-performance GPU can face a similar problem.
So when people design large AI and HPC systems, they aren’t only asking:
How fast can my GPUs calculate?
They also have to ask:
How quickly can I feed those GPUs?
and:
How quickly can those GPUs exchange data with one another?
To answer that, we need two concepts. Bandwidth and latency.
They sound similar.
Bandwidth: How much can I move?
Let’s build a highway with one lane.
Now let’s build eight lanes.
Which road can transport more cars during the same period?
Obviously the second.
That’s a useful way to think about bandwidth.
Bandwidth tells us how much data a communication link can theoretically transport over a period of time.
You will encounter numbers such as:
10 Gb/s
25 Gb/s
100 Gb/s
200 Gb/s
400 Gb/s
800 Gb/sGb/s means gigabits per second.
Don’t worry about specific InfiniBand speeds or generations yet. We’ll devote an entire article to the physical side of the fabric later.
For now, think:
Bandwidth = How wide is my data highway?
But there’s a trap here.
Suppose I build you a 50-lane highway.
Amazing bandwidth.
Unfortunately, the highway starts 200 miles away from your house.
You want to send one tiny package to your neighbor.
Is the 50-lane highway automatically fast for that task?
Not necessarily.
That brings us to latency.
Latency: How long do I wait?
Suppose Server A needs to send a small piece of information to Server B.
Server A Server B
| |
| ----------- information ----------> |
| |Latency is concerned with the delay involved in that communication.
In computing we may talk about:
milliseconds = 10⁻³ seconds
microseconds = 10⁻⁶ seconds
nanoseconds = 10⁻⁹ secondsAt human scale, microseconds sound absurdly small.
Who cares about a few microseconds?
Computers do.
Especially when the operation doesn’t happen once.
Imagine paying a tiny communication penalty millions of times.
Small delays accumulate.
And in tightly synchronized workloads, one participant may be waiting for another.
Get Jaya Sandeep Ketha’s stories in your inbox
Join Medium for free to get updates from this writer.
This is why some high-performance applications care deeply about latency.
So now we have two different questions:
Bandwidth:
How MUCH data can I move?
Latency:
How LONG does communication take?And AI/HPC often wants both:
AI / HPC Network
|
+------+------+
| |
v v
High Bandwidth Low Latency“But if I have huge bandwidth, why should I care about latency?”
Great question.
Imagine I operate two delivery companies.
Company A owns 100 trucks. Once everything starts moving, it can transport an enormous amount of cargo.
But every order takes 30 minutes before the first truck leaves.
Company B also has 100 trucks, but dispatch happens almost immediately.
If you’re moving an entire warehouse, both companies may spend most of their time transporting cargo.
But what if you need to send thousands of tiny, time-sensitive packages throughout the day?
That dispatch delay suddenly matters a lot.
Networking has a similar distinction.
For a huge continuous transfer: bandwidth becomes extremely important.
For frequent small communications: latency can become extremely important.
AI/HPC workloads can involve both.
They may move enormous amounts of data and communicate frequently.
That’s why simply advertising a large bandwidth number doesn’t tell us everything about how a network will behave for a workload.
Later in this series, when we actually benchmark RDMA, we’ll revisit this distinction. You’ll see why performance tests often measure bandwidth and latency separately, and why message size matters.
But we need another piece of the puzzle first.
What does a distributed workload actually feel like?
Forget the details of AI training for a moment.
Imagine a workload doing this:
COMPUTE
|
v
COMMUNICATE
|
v
COMPUTE
|
v
COMMUNICATE
|
v
COMPUTEIf communication is quick:
COMPUTE ███████████████
NETWORK ██
COMPUTE ███████████████
NETWORK ██
COMPUTE ███████████████Most of our time is spent doing useful computation.
Now imagine the network is slow:
COMPUTE ███████████████
WAIT ████████████████████
COMPUTE ███████████████
WAIT ████████████████████The GPUs didn’t suddenly become slower.
They’re still capable of exactly the same computation.
But the system became slower.
That’s the difference between component performance and system performance.
And it leads to an interesting consequence.
As compute becomes faster, networking can actually become more important, not less.
If cooking takes 10 minutes and delivery takes 30 seconds, delivery isn’t a huge concern.
Improve cooking to 30 seconds while delivery remains 30 seconds, and suddenly half the workflow is communication.
So improving one part of a system can expose bottlenecks somewhere else.
That is classic systems engineering.
The network is now on the critical path
Let’s simplify our workload to:
GPU COMPUTE
|
v
COMMUNICATION
|
v
GPU COMPUTE
|
v
COMMUNICATION
|
v
GPU COMPUTEIf the second compute stage cannot begin until communication finishes, then network time is part of the application’s critical path.
A rough mental model might be:
Total Workload Time
≈
Compute Time
+
Communication Time
+
Data Movement / Storage Time
+
Synchronization & Software OverheadThis isn’t intended as a universal performance equation.
It’s a way to change how we think.
A cluster isn’t fast because one component is fast.
A cluster is fast when the components can work together efficiently.
And now we’ve reached the question that eventually leads us to InfiniBand.
Wait… don’t we already have Ethernet?
Exactly.
We already have an extraordinarily successful networking technology.
Ethernet.
It connects servers, workstations, datacenters, switches, routers, storage systems, cloud infrastructure and an enormous portion of the technology around us.
And TCP/IP has proven that it can scale to something much larger than an AI cluster:
the Internet.
So if our problem is:
“Server A needs to communicate with Server B”
why can’t we simply do what computers have been doing for decades?
Application
|
v
Operating System
|
v
TCP/IP
|
v
NIC
|
v
Ethernet
|
v
NetworkWe absolutely can.
And we do.
But here’s where things get interesting.
The Internet and an HPC/AI compute fabric were built around very different problems.
Think about what we care about in a tightly coupled compute cluster:
How much CPU work is required to move data?
How many times does that data get copied?
Does the operating system need to participate in every communication?
What happens when network buffers fill?
Who guarantees reliable delivery?
How is congestion handled?
How quickly can one machine access data associated with another?
Suddenly, “How many gigabits per second?” is only one piece of the problem.
And this is where the architecture becomes fascinating.
The material we’ll build from later contrasts traditional TCP/IP and InfiniBand across exactly these dimensions: sockets versus Queue Pairs, software versus hardware-assisted reliability, copy-based versus zero-copy data movement, different approaches to flow control, and different approaches to fabric routing and management.
But I’m deliberately not explaining any of those yet.
Because if I tell you that InfiniBand uses Queue Pairs and RDMA before we’ve understood the problem they solve, they’re just terms to memorize.
I don’t want you memorizing them.
I want them to feel inevitable when we reach them.
For much of this series, we’re going to keep asking the same deceptively simple question:
How can we move data through this path with as little delay, CPU involvement and unnecessary work as possible?
We’ll gradually replace those generic boxes with real technologies.
Eventually, our picture will look more like:
AI Application
|
v
NCCL
|
v
GPU Memory
|
v
GPUDirect RDMA
|
v
ConnectX HCA
|
v
InfiniBand
|
v
Quantum Switch
|
v
Remote ConnectX HCA
|
v
Remote GPU MemoryRight now, most of those words may mean nothing to you.
That’s fine.
By the end of this series, they won’t.
More importantly, you should be able to look at that path and reason about it:
Where is the bottleneck?
Which component owns this operation?
Why is this link down?
Why isn’t this port ACTIVE?
Why is RDMA bandwidth below expectation?
Why are the GPUs communicating through an inefficient path?
Why isn’t a cluster achieving the performance its hardware suggests it should?
That’s the level I want this series to reach.
Not just “What is InfiniBand?”
But:
How does the whole thing actually work?
Where do we go next?
We started with a simple computer.
Then we added another computer.
That forced us to communicate.
We scaled to many GPUs, and communication became part of our workload.
That introduced bandwidth.
Then latency.
Then synchronization.
Then the realization that the network itself can sit on the critical path of distributed computation.
And finally we arrived at a very reasonable question:
If Ethernet and TCP/IP already connect the world, what exactly needed to change for high-performance computing?
That’s where Part 2 begins.
In the next article, we’re going to follow a piece of data through a conventional network.
Not conceptually.
Step by step.
Application Memory
|
v
Socket
|
v
Kernel
|
v
TCP/IP
|
v
NIC Driver
|
v
NIC
|
v
Network
|
v
Remote MachineWe’ll look at what the CPU is doing, where the operating system gets involved, where data moves, why copies matter, how reliability works, and what “network overhead” actually means.
Then we’ll finally be in a position to ask:
What would we design differently if our priority wasn’t connecting billions of general-purpose devices, but connecting extremely powerful computers that need to exchange data as quickly and efficiently as possible?
And from that question, InfiniBand starts to make sense.
Next: Part 2 - Ethernet/TCP/IP vs. InfiniBand: Why High-Performance Computing Took a Different Path
Thanks for reading!
If you made it this far, thank you for spending your time with the first article in this series.
My goal with AI Networking from the Wire to the GPU is to build a practical, beginner-friendly path into InfiniBand, RDMA, and NVIDIA networking - starting with why these technologies exist and gradually working our way toward how real AI/HPC infrastructure is designed, operated, and troubleshot.
I’m learning and exploring this space deeply as I write, so I’d genuinely love to hear from you.
If you found this useful, follow me to continue the series. And please leave a comment with your thoughts, corrections, questions, or ideas for topics you’d like me to cover. Discussions and feedback not only make the series better - they also keep me motivated to continue building it.
Thank you for reading, and I’ll see you in Part 2!
