AI Training Checkpoint Storage: Why 128,000-GPU Clusters Break the Storage Stack

GPU failure rates are rising exponentially with cluster size, turning checkpoint write performance into a first-order constraint on whether frontier training runs finish at all.

Checkpointing is the third storage access pattern in AI infrastructure, and it is the one the industry did not build for. Training data ingestion is read-optimized. KV cache is latency-optimized. Checkpointing is write-burst-optimized, and no existing storage tier serves that profile well. Here is what the failure math looks like at frontier scale, and what it implies for storage architecture.

• Checkpoints are large and frequent. A 70-billion-parameter model produces a 1 to 2 terabyte checkpoint. A trillion-parameter MoE model can exceed 10 terabytes. Intervals run from 30 minutes to a few hours.
• Mean time between failures collapses as clusters scale. Roughly 3 hours at 16,000 GPUs, about 90 minutes at 32,000, and a projected 14 minutes at 128,000.
• The cost is measured in lost training time. At 128K-GPU scale, failures can slow training progress by up to 43 percent. ByteDance reports failover events already exceeding 10 minutes at 10,000-GPU scale.
• Four research approaches are competing. Asynchronous checkpointing, in-memory checkpointing, delta and compressed checkpointing, and checkpoint-free fault tolerance.
• The requirements conflict with existing tiers. Checkpoint storage needs burst write absorption, durability, and fast sequential read for recovery. Parallel file systems and CXL memory pools each deliver only part of that.
• Tiered checkpoint storage is the emerging best practice. A fast staging tier absorbs the burst, then flushes to durable storage in the background without blocking training.
• The purpose-built tier does not exist yet as a product category. The failure rate math at 128K-GPU scale makes it economically inevitable.

Three Access Patterns, and the Industry Planned for Two

There are three distinct storage access patterns in AI infrastructure, and the industry planned for only two of them.

Training data ingestion is the first: read-heavy, sequential, large files, high throughput, well understood. KV cache for inference is the second, the one we covered in our analyses of the STX storage architecture and CXL 4.0 memory pooling, and it is being addressed by a wave of purpose-built systems.

The third access pattern is checkpointing, and it is the one that catches infrastructure teams off guard.

What a Training Checkpoint Actually Contains

A training checkpoint captures the complete state of a distributed training job: model weights, optimizer state (momentum, variance accumulators), learning rate schedules, dataloader positions, random number generator states, and parallelism metadata.

For a 70-billion-parameter model, a single full checkpoint is roughly 1 to 2 terabytes. For a trillion-parameter MoE model, it can exceed 10 terabytes.

The access pattern is write-heavy, bursty (hundreds of GPUs writing simultaneously at checkpoint time), latency-tolerant in absolute terms but critically time-sensitive relative to GPU idle cost, and it repeats at intervals ranging from every 30 minutes to every few hours.

At a 128,000-GPU cluster running a frontier training job with hourly checkpoints, the storage system must absorb multi-terabyte bursts at hundreds of gigabytes per second, then sit quiet until the next burst. No other workload in the data center looks like this. Not file serving, not object storage, not database I/O, not even the KV cache patterns we described earlier.

And the economic pressure driving checkpoint frequency is intensifying faster than storage architectures can adapt, because the failure rate of large GPU clusters is rising exponentially with scale.

Figure 1 — Three storage access patterns in AI infrastructure. Training data ingestion (well-understood, solved) and KV cache (emerging, STX/CXL addressing it) are the two patterns the industry planned for. Checkpointing — write-heavy, bursty, terabyte-scale, GPU-cost-sensitive — is the third pattern, and it is growing faster than storage architectures can adapt. A single 70B model checkpoint includes weights, optimizer state (3× weights for Adam), gradients, and metadata, totaling 700 GB–1.3 TB.

The Failure Math: MTBF Collapses as Clusters Grow

The economics of checkpointing are driven by failure rates, and failure rates at scale are worse than most planning assumptions account for.

Meta’s Llama 3 training paper provides the canonical data: on a 16,000-GPU cluster running for 54 days, they observed 419 failures, approximately 7.76 failures per day. MLCommons’ MLPerf Storage Working Group built a scaling model from this data that shows mean time between failures (MTBF) decreasing exponentially with cluster size, because the majority of failure modes scale linearly with the number of components.

At 16,000 GPUs, MTBF is roughly 3 hours. At 32,000, it drops to approximately 90 minutes. At 128,000 GPUs, the scale that Meta’s Prometheus clusters and frontier training runs are targeting in 2026, the projected MTBF falls to approximately 14 minutes.

Fourteen Minutes

Fourteen minutes. That means on a 128K-GPU cluster, you can statistically expect a failure before a one-hour checkpoint interval completes, and you will lose all training progress since the last successful checkpoint.

If each checkpoint takes 10 minutes to write and failures arrive every 14 minutes on average, the system spends more time writing checkpoints and recovering from failures than it does training. The Effective Training Time Ratio (ETTR), the fraction of wall-clock time spent on productive training, collapses.

ByteDance reported that failover operations at 10,000-GPU scale already consume more than 10 minutes per event, including machine rescheduling, environment reconstruction, checkpoint loading, and recomputation of lost progress. At 128K-GPU scale, failures can slow training progress by up to 43 percent.

The implication is that checkpoint storage performance is not a nice-to-have optimization. It is a first-order determinant of whether large-scale training runs can complete at all, and the storage system that handles checkpoints must be purpose-built for a workload profile that no existing storage tier was designed to serve.

Figure 2 — Mean time between failures versus cluster size. Based on Meta’s Llama 3 data (419 failures over 54 days on 16K GPUs), MTBF decreases exponentially as cluster size grows. At 128K GPUs, MTBF drops to approximately 14 minutes — below the typical 10-minute checkpoint write time. When failures arrive faster than checkpoints complete, traditional periodic checkpointing breaks down entirely.

Four Approaches to the Checkpoint Bottleneck

The academic and industry response has been a burst of systems research that accelerated sharply in 2024 and 2025. The approaches divide into four categories, each attacking a different dimension of the problem.

1. Asynchronous Checkpointing

Instead of pausing training while the checkpoint writes to storage, the system copies the training state to CPU memory (or a staging buffer) and continues training while the state is flushed to persistent storage in the background.

ByteDance’s ByteCheckpoint (NSDI 2025) is the most production-hardened example, deployed at scale across ByteDance’s GPU clusters with a parallelism-agnostic checkpoint representation that decouples the saved format from the training parallelism configuration.

Get Brandon Mincey’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

This is critical for elastic training, where the parallelism may change between save and load. A training run that saved on 256 GPUs with 8-way tensor parallelism may need to reload on 192 GPUs with 6-way tensor parallelism after a node failure. ByteCheckpoint handles this resharding automatically at load time, eliminating the manual reconfiguration that previously consumed hours of engineer time per failure event.

2. In-Memory Checkpointing

Exemplified by Gemini, this approach stores checkpoints in CPU memory of neighboring nodes rather than writing to persistent storage. It enables high-frequency saves at sub-minute intervals with minimal latency, at the cost of using significant CPU memory capacity.

3. Delta and Compressed Checkpointing

Check-N-Run uses differential saves that capture only parameters that changed since the last checkpoint. The Inshrinkerator and ExCP systems apply dynamic quantization and weight-momentum joint compression to shrink checkpoint size by 2 to 4 times.

4. Checkpoint-Free Fault Tolerance

The most radical category. FlashRecovery demonstrated training restoration on a 4,800-device cluster in 150 seconds by using redundant computation and in-memory state replication to eliminate checkpointing entirely. Lazarus, built for MoE models, achieves up to 5.7 times improvement over checkpoint-based systems under frequent failures by building resilience directly into the pipeline parallelism with elastic expert placement.

The Benchmark Signal

MLCommons recognized the urgency by releasing the MLPerf Storage v2.0 checkpointing benchmark in August 2025. It is the first standardized benchmark specifically measuring checkpoint write and recovery performance, a signal that the industry now treats checkpoint storage as a distinct, measurable infrastructure capability.

Figure 3 — Four approaches to the checkpoint bottleneck. Asynchronous checkpointing decouples saves from training. In-memory checkpointing uses neighbor node DRAM for sub-minute intervals. Delta and compressed checkpointing reduces write volume by 2–4×. Checkpoint-free fault tolerance eliminates checkpointing entirely through redundant computation and elastic pipeline adaptation. Most production systems combine multiple approaches.

Three I/O Profiles, One File System, Worst of All Worlds

The storage architecture implications are concrete and underappreciated. Checkpoint storage has requirements that conflict with both training data storage and KV cache storage.

• Training data storage is read-optimized. Parallel file systems like Lustre and GPFS are tuned for high-throughput sequential reads from many clients.
• KV cache storage (CMX, CXL pools) is latency-optimized. Nanosecond to microsecond access for small, random, ephemeral data with no durability.
• Checkpoint storage is write-burst-optimized. It needs to absorb hundreds of gigabytes per second of concurrent writes from every GPU in the cluster for a brief window, then go idle. It must also provide durability, because you cannot afford to lose a checkpoint, and fast sequential read performance for recovery, because you need to reload terabytes when a failure occurs.

These are three fundamentally different I/O profiles. Trying to serve all three from a single parallel file system leads to the worst-of-all-worlds outcome: checkpoint writes contend with training data reads, both suffer performance degradation, and the storage system is either over-provisioned for steady-state training I/O (wasting capacity) or under-provisioned for checkpoint bursts (causing GPU stalls).

The Emerging Best Practice: Tiered Checkpoint Storage

A fast staging tier (in-memory, CXL, or local NVMe) absorbs the burst write at GPU speed, then a background process flushes the staged data to durable storage (parallel file system, object store, or cloud storage) without blocking training.

AWS documented this hierarchical approach in detail, recommending S3 Express One Zone or FSx for Lustre as the durable tier with local NVMe or instance memory as the staging tier.

The STX architecture we analyzed in Blog 5 adds another option: BlueField-4 DPU-managed NVMe could serve as the staging tier for checkpoint bursts, with the DPU handling the background flush to durable G4 storage. NVIDIA has not positioned CMX for this use case yet, since CMX is explicitly optimized for KV cache, which has different durability requirements.

The Product Category That Does Not Exist Yet

The gap in the market is a purpose-built checkpoint storage tier that combines burst write absorption, durability, fast sequential read for recovery, and automatic resharding support.

That tier does not exist as a product category today. The failure rate math at 128K-GPU scale makes it economically inevitable.

Figure 4 — Checkpoint storage in the AI infrastructure stack. The training loop periodically emits burst writes into a fast staging tier (CPU memory, CXL, local NVMe) that absorbs the data without blocking training. A background process flushes staged checkpoints to durable storage. Recovery reads the durable tier at high throughput to reload state after failures. The three AI storage I/O profiles — training data (read-heavy), KV cache (random, ephemeral), and checkpoints (write-burst, durable) — have fundamentally different requirements and should not share the same storage tier.

Sources:

• MLCommons “MLPerf Storage v2.0 Checkpointing Workload” (August 2025)
• Meta “Llama 3: Herd of Models” (2024)
• 419 failures / 54 days / 16K GPUs failure data
• Wan et al. “ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development” (NSDI 2025)
• ByteDance “ByteRobust: Robust LLM Training Infrastructure” (2025)
• FlashRecovery “Fast and Low-Cost Recovery from Failures for Large-Scale Training of LLMs” (2025)
• Lazarus “Resilient and Elastic Training of Mixture-of-Experts Models” (2025) · AWS “Architecting Scalable Checkpoint Storage for Large-Scale ML Training” (June 2025)
• CUDO Compute “How Checkpointing Impacts AI Infrastructure Storage Requirements” (September 2025)
• Nebius “Fault-tolerant Training: How We Build Reliable Clusters” (August 2025)
• Gandhi & Kozyrakis “JIT-Checkpointing” (EuroSys 2024)
• ExCP (ICML 2024)
• Inshrinkerator (SoCC 2024)
• PCcheck (ASPLOS 2025)
• FlowCheck (EuroSys 2025)