GPU Cloud Benchmarks 2026: AI GPU Throughput, Specs, Pricing

Updated April 2026 with live Spheron pricing, B200 and B300 Blackwell data, and refreshed throughput benchmarks from MLPerf Inference v6.0.

Choosing a GPU for your AI workload shouldn't require a spreadsheet and three hours of tab-hopping across provider websites. But that's exactly what it takes today, because no provider publishes honest side-by-side comparisons. Each one highlights the metric where they win and buries the rest. This guide helps teams navigate the landscape; for specific selection guidance, see our top 10 cloud GPU providers analysis.

This post fixes that. It's a working set of AI GPU benchmarks plus current pricing across the providers that matter most in 2026. Real specs, published inference throughput data, no marketing spin, just the numbers you need to make an informed decision. For a workload-based decision guide that maps these GPUs to specific inference scenarios, see best GPU for AI inference in 2026. If you're still getting oriented on what a GPU cloud actually is and how billing works before diving into benchmark tables, our what is a GPU cloud primer covers that ground first.

For a provider-by-provider pricing breakdown across all GPU models, see our GPU cloud pricing comparison for 2026.

TL;DR: GPU Cloud Benchmarks and Pricing in 2026

Rates are per GPU on Spheron as of 04 Sep 2026. Memory bandwidth predicts inference throughput better than raw FLOPS on most LLM workloads, so read the first two columns together. Compare live pricing

The Hardware Landscape: What You're Actually Choosing Between

Before comparing providers, you need to understand what separates the GPUs themselves. The differences aren't just about raw compute power; memory capacity, memory bandwidth, and interconnect speed all determine whether a GPU is right for your workload.

GPU Specifications at a Glance

FP16 and FP8 Tensor Core values are dense (non-sparse). With NVIDIA 2:4 structured sparsity: FP16: A100 ~624, H100/H200 ~1,979, L40S ~733, RTX 4090 ~330 TFLOPS; FP8: H100/H200 ~3,958, L40S ~1,466, B200 ~9,000, RTX 4090 ~1,320 TFLOPS.

¹ A100 (Ampere architecture) has no native FP8 hardware support. FP8 Tensor Cores were introduced with Hopper (H100/H200) and Ada Lovelace (L40S, RTX 4090) architectures.

A few things jump out from this table.

Memory bandwidth is the inference bottleneck. For LLM inference, the speed at which you can read model weights from memory matters more than raw FLOPS. The H200's 4.8 TB/s bandwidth is why it outperforms the H100 on inference despite having identical compute; it can feed the tensor cores faster. The B200 doubles that again to 8 TB/s. Teams making this choice should read our GPU cost optimization playbook for detailed TCO analysis.

VRAM determines what you can run. The H200's 141 GB lets you run 70B parameter models in FP16 on a single GPU. The A100 at 80 GB requires quantization or multi-GPU setups for the same model. The B200's 192 GB opens the door to running 100B+ models without sharding.

The L40S is the sleeper pick for inference. Its 1,466 FP8 TFLOPS (Ada Lovelace FP8 Tensor Cores) makes it surprisingly competitive for inference workloads that can use FP8 precision, at a fraction of the cost of an H100.

Pricing Across Providers: The Real Numbers

GPU pricing varies enormously depending on which provider you use, whether you commit to a reservation, and whether you're willing to use spot instances. Here's what the market looks like right now.

H100 Pricing (per GPU, per hour)

Pricing fluctuates based on GPU availability. Spheron rates above are live as of 04 Sep 2026; other providers reflect their own published rates, checked on 15 Apr 2026, and may have changed. Check current GPU pricing → for live rates.

The spread is significant: H100 on Spheron at $2.65/hr is a fraction of the AWS and Azure rates above. That's the same GPU, the same CUDA cores, the same 80 GB of HBM3; just a different billing address.

Two years ago, H100s commanded $8/hr on-demand. The market commoditized rapidly through 2025 and into 2026 as supply caught up. Surveying the published rate cards above on 15 Apr 2026, on-demand pricing had fully bifurcated: specialist clouds and marketplaces sat at $2-3/hr, hyperscalers at $3-7/hr for the same silicon.

H200 SXM Pricing

Pricing fluctuates based on GPU availability. Spheron rates above are live as of 04 Sep 2026; other providers reflect their own published rates, checked on 15 Apr 2026, and may have changed. Check current GPU pricing → for live rates.

The H200 carries a 10-20% premium over the H100 SXM at most specialist clouds, justified by its 42% inference throughput improvement. For latency-sensitive inference, cost-per-token math often favors H200 despite the higher hourly rate.

One catch: hyperscalers still sell H200s only in 8-GPU bundles. If you need a single H200, specialist clouds like Spheron are your only real option for per-GPU billing. Note that "H200" also splits into two distinct form factors with different NVLink topology and rack requirements; our H200 NVL vs SXM5 decision guide covers which one actually fits your deployment before you commit to a rate above.

A100 80GB Pricing

Pricing fluctuates based on GPU availability. Spheron rates above are live as of 04 Sep 2026; other providers reflect their own published rates, checked on 15 Apr 2026, and may have changed. Check current GPU pricing → for live rates.

The A100 is in its sunset phase for premium workloads, but still relevant for mid-tier cost-sensitive training and inference. A100 80G SXM4 on Spheron at $1.43/hr on-demand (or $1.19/hr spot) is now a solid choice for any workload that fits in 80 GB. Fine-tuning models under 30B parameters, batch inference, and lighter training jobs don't need Hopper or Blackwell class hardware. See our A100 vs V100 comparison for deeper context on when A100 is still the right call.

L40S Pricing

The L40S sits in a useful price-performance pocket for inference. On Spheron it is $0.96/hr on-demand and $1.07/hr spot as of 04 Sep 2026, and the third-party rates above ran $0.40-$0.86/hr when checked on 15 Apr 2026. With 48 GB VRAM and strong FP8 throughput from its Ada Lovelace Tensor Cores, it handles most production inference workloads well below H100 pricing.

For full L40S benchmark data and workload guidance, see our L40S for AI inference guide.

B200 and B300 Pricing

The most significant pricing shift in Q1-Q2 2026 is Blackwell. B200 was in short supply through late 2025, with providers quoting $6-8/hr on their own rate cards when they had inventory. By April 2026, supply caught up and rates collapsed.

Pricing fluctuates based on GPU availability. Spheron rates above are live as of 04 Sep 2026; other providers reflect their own published rates, checked on 15 Apr 2026, and may have changed. Check current GPU pricing → for live rates.

B200 SXM6 is $9.36/hr on-demand and $5.37/hr spot, for 2.4x the memory bandwidth and 2.4x the VRAM (192 GB vs 80 GB) of H100, with native FP4 support. Compare both against H100 on-demand at $2.65/hr before defaulting to Hopper; the pools move independently, so check the live figures rather than carrying over a remembered gap. B200 spot suits fault-tolerant inference that can absorb a reclaim. See our H200 vs B200 vs GB200 guide for detailed benchmarks.

B300 (Blackwell Ultra) is $9.16/hr on-demand and $4.76/hr on spot as of 04 Sep 2026. If your training run has checkpointing, B300 spot on Spheron gives you 288 GB HBM3e and 15 PFLOPS dense FP4 at the live rate above. Full breakdown in our B300 Blackwell Ultra guide.

What Changed Between Q1 and Q2 2026

If you read the earlier version of this post in March, a few things have shifted enough to call out explicitly:

1. B200 has both tiers quoted again. B200 SXM6 is $9.36/hr on-demand and $5.37/hr spot, against H100 at $2.65/hr on-demand, for 2.4x the bandwidth and VRAM. Check every tier against the others before assuming Hopper is cheaper. B200 spot suits checkpoint-capable workloads.
2. B300 spot is quoted for interruptible training. $4.76/hr buys 288 GB HBM3e. If your training stack handles preemption cleanly, compare it against H100 at $2.65/hr before you assume you need 3.6x the memory.
3. AWS p5e bumped H200 up 15% in January 2026 while specialist clouds held flat. The hyperscaler gap widened, not narrowed.
4. A100 80G SXM4 is $1.43/hr on-demand and $1.19/hr spot as of 04 Sep 2026. Still affordable for sub-30B fine-tuning and mid-tier training.
5. L40S pricing stayed flat in the $0.80-$0.86/hr range on specialist-cloud rate cards checked on 15 Apr 2026. Ada-gen FP8 throughput has kept it competitive for 7B-30B inference even as Blackwell ramped.

For pricing moves and hardware launches that land between our quarterly refreshes here, our weekly GPU cloud news digest tracks them as they happen.

AI GPU Benchmarks: Inference Throughput Where It Actually Matters

Raw specs and pricing only tell half the story. What matters is how many tokens per second you get for your dollar. Here's published benchmark data from MLPerf inference benchmarks on Llama 70B.

Llama 70B Inference Throughput

The node rates in that table are the ones live on 15 Apr 2026 and are held fixed, because the cost-per-million column is derived from them; changing one number without the others would make the ratios wrong. They are not current prices: for those, see the live per-GPU rates in the pricing tables above.

Methodology note: The A100 figure is a single-GPU result running Llama 2 70B with INT8 quantization (the model barely fits in 80 GB quantized). The H100/H200 figures are from MLPerf Inference v4.0 8-GPU server submissions using TensorRT-LLM with FP8. These are not equivalent configurations; comparing 1× A100 to 8× H100 inflates the ratio. On a per-GPU basis with the same precision, H100 SXM outperforms A100 by roughly 5-8× for Llama 2 70B throughput. The multi-GPU aggregate numbers are shown here because they reflect real deployed configurations. The $/hr column shows the full 8-GPU node rate (8 × per-GPU price); cost-per-million-tokens divides that node rate by the aggregate throughput in tokens per hour.

For a breakdown of the latest MLPerf Inference v6.0 scores and what they mean for cloud GPU selection, see our MLPerf v6.0 results guide.

For smaller models (7B-13B parameters), the gap narrows. An A100 runs Llama 3.3 8B at roughly 400-500 tok/s, while an H100 pushes 10,000+ tok/s with TensorRT-LLM. The cost-per-token advantage of newer hardware remains, but the absolute throughput is less dramatic.

Practical Inference Scenarios

Chatbot / real-time API (latency-sensitive): H200 is the best choice. The ~42% throughput improvement over H100 directly translates to lower latency at high concurrency. The 141 GB VRAM also lets you run larger models without quantization, preserving output quality. Understand the tradeoffs in our detailed nvidia-h100-vs-h200 comparison.

Batch inference / offline processing: H100 at marketplace pricing ($1.49-$2.10/hr on host listings checked on 15 Apr 2026) offers strong cost-per-token. Latency doesn't matter for batch jobs, so you're purely optimizing for throughput per dollar.

Budget inference / low-traffic endpoints: L40S or A100. If your model fits in 48 GB (L40S) or 80 GB (A100), the L40S is $0.96/hr on-demand on Spheron and third-party rates ran $0.40-$0.86/hr when checked on 15 Apr 2026. For startups running a single model endpoint with moderate traffic, this is 3-5x cheaper than an H100 and more than sufficient.

Maximum throughput (large-scale serving): B200 when pricing normalizes. The 8 TB/s bandwidth and native FP4 support promise 11-15x throughput improvements over Hopper-generation GPUs. If you're serving millions of requests per day, the B200's higher hourly cost is offset by dramatically fewer GPUs needed.

Training Performance: How Much Faster Is Your Model Ready?

Training benchmarks are harder to standardize because they depend heavily on model architecture, batch size, precision, and optimization framework. But the relative performance between GPU generations is consistent.

Training Speedup by Generation

The H100's 2.4x training speedup over A100 translates directly to cost savings: a training run that takes 10 days on A100s finishes in roughly 4 days on H100s. Even though H100s cost 3x more per hour, the total training cost is lower because you're paying for fewer hours.

For distributed training across multiple nodes, interconnect matters enormously. NVLink bandwidth determines how fast gradients synchronize between GPUs. The B200's NVLink 5 at 1.8 TB/s per GPU is 2x the H100's NVLink 4; this means less time waiting for communication and more time computing.

The Decision Framework

With all this data, here's the practical framework for choosing a GPU and provider.

Step 1: What's your workload?

• Inference (real-time) → Optimize for latency. H200 or H100.
• Inference (batch) → Optimize for cost-per-token. H100 marketplace or L40S.
• Training (fine-tuning) → Optimize for VRAM + cost. A100 or H100 depending on model size.
• Training (pre-training) → Optimize for sustained throughput. H100/H200 clusters with InfiniBand.

Step 2: What's your budget tolerance?

• Minimum cost, flexible on availability → GPU marketplaces and spot instances.
• Predictable pricing, guaranteed availability → Reserved instances from mid-tier providers.
• Maximum reliability, budget secondary → Hyperscalers (AWS, GCP, Azure).
• Small, latency-tolerant models on a tight budget → older or consumer-grade cards; see our A10G vs T4 pricing breakdown and, on the Blackwell consumer side, the RTX 5080 vs RTX 5090 rental cost comparison for where cheaper hardware still clears the bar.

Step 3: Compare cost-per-output, not cost-per-hour.

A GPU that costs 2x more per hour but delivers 3x the throughput is 33% cheaper on a per-output basis. Always calculate cost per million tokens (inference) or cost per training epoch (training), not just the hourly rate. For a detailed token factory benchmarks table with live Spheron pricing across A100, H100, H200, and B200, see the token factory benchmarks guide.

What We'd Recommend

For teams evaluating GPU cloud options in April 2026, the market has shifted hard in your favor. H100 pricing has dropped 75% from its 2023 peak, A100s are at $1.43/hr on-demand or $1.19/hr spot on Spheron, B200 spot is $5.37/hr for 2.4x the bandwidth, and B300 spot is $4.76/hr. Compare each tier against H100 at $2.65/hr rather than assuming an ordering: the pools move independently. Learn which GPU suits your specific model by consulting our best NVIDIA GPUs for LLMs guide and checking current pricing.

The biggest remaining challenge isn't finding cheap GPUs; it's comparing across providers without spending a full day on it. Spheron aggregates pricing across 5+ providers into a single interface, making it practical to consistently deploy on the cheapest available option for your workload. Whether you need spot H100s for a training run or reserved L40S instances for a production endpoint, you're choosing from the full market instead of one provider's pricing.

We'll update this benchmark data quarterly as new hardware ships and pricing evolves. Bookmark this page if you want the latest numbers without the research.

Spheron aggregates GPU pricing from vetted data center partners into a single interface, so you can deploy on the cheapest available option for your workload. As of 04 Sep 2026: A100 from $1.43/hr on-demand ($1.19/hr spot), H100 from $2.65/hr, B200 spot from $5.37/hr, B300 spot from $4.76/hr. No contracts, per-minute billing.

Spheron H100 → | B200 GPU pricing → | View all pricing →