Sitemap
techbeatly

Articles and how-to’s on #AI #Automation #Containerization and other #devops technologies. https://www.techbeatly.com

NVLink

7 min readMay 24, 2026

--

The hidden bottleneck that determines whether your AI system scales — or stalls

Published for: MLOPs, LLMOps, AI Engineers, Solution Architectures, AI Infrastructure

Imagine you hired 72 of the world’s fastest workers and put them in the same building. They’re brilliant. They work fast. But there’s one problem — they can only communicate by shouting across the hallway.

That’s what multi-GPU AI systems looked like before NVLink.

GPUs are powerful. But when multiple GPUs need to work together on a large AI model, they constantly need to share data with each other. The speed of that communication determines whether the system scales — or whether all that expensive compute sits idle, waiting.

NVLink is NVIDIA’s solution to that communication problem.

In this post, I’ll walk you through what NVLink is, why it matters, how it evolved, and what it means for anyone building or running AI systems today.

The Problem: GPUs Are Powerful but Isolated

Every GPU has its own memory — called HBM (High Bandwidth Memory). When you’re training a large language model across 8 GPUs, those GPUs need to constantly share:

  • Gradients (what the model learned in each training step)
  • Activations (intermediate results during forward pass)
  • Model weight shards (pieces of a model too large to fit in one GPU)

Before NVLink, all of this data moved over PCIe — the standard interface used for connecting components inside a server.

PCIe Gen 4 delivers about 32 GB/s. That sounds fast. But when you’re running a 70B-parameter model, a single inference request can generate 20 GB of synchronization data that must be exchanged between every GPU — for every request, every layer, every step.

At that scale, PCIe becomes the bottleneck. GPUs go idle waiting for data. The system underperforms. More GPUs make it worse, not better.

The Solution: NVLink

NVLink is NVIDIA’s high-speed, direct GPU-to-GPU interconnect.

Instead of sending data through the server’s PCIe bus and bouncing it through the CPU, NVLink creates a direct, dedicated highway between GPUs. Data moves faster, with lower latency, and without interrupting the CPU.

Three things make NVLink fundamentally different from PCIe:

1. Massive bandwidth NVLink 5 (the current generation in Blackwell GPUs) delivers 1,800 GB/s per GPU — that’s 28× more bandwidth than PCIe Gen 4, and 14× more than PCIe Gen 5.

2. Peer memory access With NVLink, GPU A can read and write directly into GPU B’s memory — no CPU involved, no data copying overhead. It’s as if both GPUs share the same memory space.

3. Works automatically with CUDA You don’t need to rewrite your code. If you have NVLink-connected GPUs and call cudaDeviceEnablePeerAccess(), CUDA automatically routes data transfers over NVLink instead of PCIe. Your existing PyTorch, JAX, or TensorFlow workloads benefit without changes.

A Simple Analogy

Think of PCIe as a city road network — shared, congested, and managed by traffic lights (the CPU).

NVLink is a private, direct expressway between two buildings. No traffic lights. No shared lanes. Full speed, both directions, at all times.

When you’re moving millions of cars per hour (gradient tensors), the difference between a shared road and a private expressway is the difference between a working system and a gridlocked one.

NVSwitch: The Traffic Exchange for Many GPUs

NVLink is great for connecting two GPUs. But what about 8? Or 72?

If you connect 8 GPUs with point-to-point NVLink cables, the bandwidth gets shared when multiple pairs need to talk simultaneously. GPU A’s conversation with GPU B eats into GPU C’s available bandwidth to GPU D.

NVSwitch solves this.

NVSwitch is a dedicated switching chip that sits between the GPUs and acts like a non-blocking crossbar — every GPU can talk to every other GPU at full NVLink speed, simultaneously, without any of them competing for bandwidth.

Without NVSwitch: Adding more GPUs splits available bandwidth.
With NVSwitch: Every GPU always gets its full NVLink bandwidth, regardless of how many others are communicating at the same time.

In an 8-GPU HGX H100 server, four NVSwitch chips provide this full all-to-all connectivity. Each NVSwitch chip has a total bidirectional bandwidth of 25.6 terabits per second.

That’s not a typo.

Six Generations in One Table

NVLink has evolved dramatically since its introduction in 2016:

| Generation | Architecture     | Year  | Bandwidth per GPU |
| ---------- | ---------------- | ----- | ----------------- |
| NVLink 1.0 | Pascal (P100) | 2016 | 80 GB/s |
| NVLink 2.0 | Volta (V100) | 2017 | 300 GB/s |
| NVLink 3.0 | Ampere (A100) | 2020 | 600 GB/s |
| NVLink 4.0 | Hopper (H100) | 2022 | 900 GB/s |
| NVLink 5.0 | Blackwell (B200) | 2024 | 1,800 GB/s |
| NVLink 6.0 | Rubin | 2025+ | 3,600 GB/s |

In 10 years, per-GPU bandwidth has grown 45×. The trajectory is not slowing down — it’s accelerating alongside the growth of AI model sizes.

Why This Matters for AI Workloads

Training Large Models

When training a model like GPT-4 or Llama 3 across hundreds of GPUs, each training step requires an All-Reduce operation — every GPU sends its gradients to every other GPU and receives the averaged result.

Get Praveenpm’s stories in your inbox

Join Medium for free to get updates from this writer.

On a PCIe-based system, this collective communication step becomes the bottleneck. On an NVLink system with NVSwitch, the all-reduce completes at near-theoretical peak bandwidth, and GPU compute cores stay busy instead of waiting.

NVIDIA’s own measurements show that an 8-GPU all-reduce for a 20 GB tensor completes in ~22ms with NVSwitch — versus ~150ms without. That’s a 7× speedup on a single operation that repeats thousands of times per training run.

Inference on Large Models

For serving large models in production — say, a 70B or 405B parameter model — the model must be split across multiple GPUs using tensor parallelism. Each GPU holds a slice of the model’s weight matrices.

During inference, after each layer, every GPU must exchange its partial results with all others before proceeding to the next layer. This happens for every single token generated, for every user request.

NVLink’s bandwidth is the direct determinant of how fast tokens appear in the user’s chat window. Faster interconnect → faster tokens → better user experience.

Long-Context and Agentic AI

Modern AI workloads involving very long contexts (128K–1M tokens) or multi-step agentic reasoning generate KV caches (key-value caches that store past attention states) that can be too large to fit in a single GPU’s memory.

NVLink enables KV caches to be distributed across the GPUs in an NVLink domain and accessed at memory speeds — making these workloads tractable on hardware that would otherwise be insufficient.

The GB200 NVL72: What NVLink Looks Like at Scale

NVIDIA’s current flagship system — the GB200 NVL72 — is the most powerful NVLink deployment in production today.

Here’s what it contains:

  • 72 Blackwell B200 GPUs, each with 1,800 GB/s NVLink 5 bandwidth
  • 36 Grace CPUs, connected to their paired GPUs via NVLink-C2C (a chip-to-chip variant)
  • 9 NVLink Switch trays providing the all-to-all fabric
  • 130 TB/s total aggregate bandwidth across the rack
  • Liquid cooling (required — the thermal density rules out air cooling)

From the software’s perspective, the entire rack looks like a single unified accelerator — one giant compute pool with shared memory. vLLM, TensorRT-LLM, and other inference frameworks treat the 72 GPUs as one coherent system.

Scaled further, 8 of these racks form a DGX SuperPOD — 576 Blackwell GPUs connected in a single NVLink domain with 1 petabyte per second of aggregate bandwidth.

NVLink vs. PCIe vs. InfiniBand

People often ask: “If I have InfiniBand, do I still need NVLink?”

Yes. They serve completely different purposes.

Technology Where It Works What It’s For NVLink Inside a rack GPU↔GPU fast communication (tensor shards, gradients) PCIe Inside a server GPU↔CPU, GPU↔storage, GPU↔network card InfiniBand / Ethernet Across racks and data centers Node↔node communication for large clusters

| Technology            | Purpose                 |
| --------------------- | ----------------------- |
| NVLink | GPU↔GPU communication |
| PCIe | GPU↔CPU / storage |
| InfiniBand / Ethernet | Rack-to-rack networking |

Every well-designed AI cluster uses all three. NVLink handles the intense, high-frequency communication within the rack. InfiniBand handles the communication between racks. PCIe handles everything connecting to the host.

Choosing between them isn’t the right question. Knowing which layer each one serves is.

What’s Next: NVLink 6 and NVLink Fusion

NVLink 6 (Rubin Architecture)

The next generation, shipping with NVIDIA’s Rubin platform, doubles NVLink 5’s bandwidth to 3,600 GB/s per GPU — 14× the bandwidth of PCIe Gen 6. Seventy-two Rubin GPUs connected in a Vera Rubin NVL72 rack will have 260 TB/s of aggregate bandwidth.

NVLink 6 also introduces in-switch computation via the SHARP protocol, where AllReduce operations are partially computed inside the NVLink Switch itself — eliminating additional round-trips to GPU memory.

NVLink Fusion

Perhaps more strategically significant: in 2025, NVIDIA opened NVLink to third-party silicon via NVLink Fusion. AWS has already integrated their Trainium4 custom chip into the NVLink fabric using this technology.

This means NVLink is evolving from an NVIDIA-exclusive technology into a standard scale-up interconnect that heterogeneous AI hardware can plug into. It’s a significant shift for anyone thinking about long-term AI infrastructure strategy.

Key Takeaways

Here’s what to remember from this post:

  • NVLink is a direct GPU-to-GPU interconnect that replaces PCIe for intra-rack GPU communication, delivering up to 1,800 GB/s per GPU in the current Blackwell generation
  • NVSwitch adds all-to-all connectivity so every GPU in a server or rack communicates at full NVLink speed simultaneously — no sharing, no contention
  • It works transparently with CUDA — frameworks like PyTorch and vLLM use it automatically via NCCL without code changes
  • NVLink, PCIe, and InfiniBand operate at different layers — they complement each other, not compete
  • The bandwidth doubles roughly every two years — NVLink 6 hits 3,600 GB/s per GPU with Rubin in 2025–2026
  • NVLink Fusion opens the technology to third-party chips, signaling its evolution into an industry-standard scale-up fabric

Tags: NVIDIA · GPU · AI Infrastructure · Machine Learning · Deep Learning · MLOps · LLMOps · Platform Engineering

--

--

Written by Praveenpm

Solution Architect at Hewlett Packard Enterprise

Published in techbeatly

Articles and how-to’s on #AI #Automation #Containerization and other #devops technologies. https://www.techbeatly.com