GLM Built Its Own Inference Infrastructure: 2 Weeks, 100K Accelerators, 3x Higher Throughput
The inference service for GLM-5.3-Flash is now running on over 100,000 Chinese-built AI accelerators. From the first successful test run to production readiness, the entire process took less than two weeks. And when it was done, end-to-end throughput hit three times the original baseline.
One of the key contributors to this project is an Infra Agent powered by GLM-5.3 itself. Z.ai’s official engineering blog walks through how it all came together. Beyond the impressive speed gains, what stands out most is who did the work: the model is now helping build the infrastructure that runs it.
Before this project, no one had deployed Chinese-built AI accelerators at this scale. These accelerators come with relatively limited memory capacity and bandwidth, and the stack needed to support everything from a new model architecture to 1M token context windows and multimodal requests. The ecosystem was immature, kernel support was incomplete, and documentation was spotty at best.
Figure 1: End-to-end throughput evolution of GLM-5.3-Flash from first successful run to production deployment
The Core Problem: End-to-End Metrics Aren’t Enough
Performance issues in inference systems almost always stem from dynamic interactions across multiple layers: kernel implementations, parallelization strategies, communication behavior, memory management, and service orchestration. If all you get is a generic alert that “throughput dropped 20%,” the agent has no way to tell which layer the problem lives in. End-to-end metrics tell you something is wrong — they just can’t tell you why.
The traditional approach relies entirely on human engineer intuition: load test results tell you what to check next, and you slowly step through checking kernel outputs, execution timelines, and communication events one by one. These scattered, human-driven workflows don’t work out of the box for an autonomous agent.
Z.ai’s solution is what they call “dense feedback”: correctness tests, runtime logs, execution traces, runtime events, microbenchmarks, and end-to-end metrics are all fed directly into the agent’s iteration loop. This framework has three core defining features:
Sufficiently granular. Every feedback point maps directly to a specific kernel, input condition, thread, execution window, and code path. A generic alert like “accuracy dropped after fusion optimization” gives you nowhere to start. But a specific note like “Input-output for this exact request diverged after this code change” lets you trace the problem straight to its source.
Low collection cost. Any question that can be answered with kernel testing or local microbenchmarks doesn’t require a full service deployment and end-to-end stress test after every small change. Short validation cycles let the agent correct its course quickly.
Objectively verifiable. Correctness is checked against a reference implementation, and performance gains are confirmed via repeatable experimental metrics. Correlation never takes the place of causal validation.
Figure 2: The Infra Agent’s dense feedback-powered optimization loop
Case Study 1: The TF32 Precision Bug
All inference performance optimization has to build on numerical correctness. First, the agent built a clear “parallel strategy → kernel implementation” mapping on top of the inference engine, so that system-level deployment configurations could be broken down into kernel-level verifiable tasks.
On the Context Parallelism (the technique that splits long sequences across multiple devices for processing) path of the KDA kernel, parallel execution requires merging state from different context shards. The core operations here are state transform merging and state updates. In the original implementation, tl.dot defaulted to TF32 precision even when all inputs were FP32. In long context windows, this tiny precision error got amplified into real problems.
Get AI Engineering’s stories in your inbox
Join Medium for free to get updates from this writer.
The fix: Both operations were explicitly set to input_precision="tf32x3", combining three TF32 tensor core operations to produce a higher-precision result. This fix has already been merged upstream into Flash Linear Attention (PR #1180).
Case Study 2: The GIL Concurrency Bottleneck
Engineers set up a clear test scenario for the agent: standalone Prefill, Prefill + KV Transfer, and standalone Decode. The acceptance criteria was explicit: the performance gap between Prefill+KV Transfer and the standalone Prefill baseline should not exceed 5%.
The agent quickly found that gap exceeded 20% in several common scenarios. Timeline analysis revealed one clear pattern: In all of these problematic scenarios, the Python-side KV Transfer execution never overlapped with DeepEP’s dispatch/combine calls.
Tracing the issue all the way to the Python/C++ boundary made the problem obvious. In DeepEP v1.2.1, neither intranode_dispatch nor intranode_combine explicitly released the Python GIL. While the CPU was waiting for the GPU to return token counts, the C++ call holding the GIL blocked the Python thread responsible for Mooncake Transfer running in the same process. Transfer task scheduling was delayed, and the underlying asynchronous capabilities of the hardware could never be utilized.
A key side-by-side comparison confirmed this diagnosis: The same version’s internode_dispatch did explicitly release the GIL, with a comment explicitly noting that the change was meant to avoid blocking KV Transfer on other threads.
The fix: Add GIL releases to the relevant C++ execution windows. After the fix, the performance gap between Prefill+KV Transfer and standalone Prefill dropped to under 1% under identical test conditions.
Figure 3: Discovery and fix of the KV Transfer concurrency bottleneck
Case Study 3: Extracting Optimizations From Existing Kernels
Decades of tuning experience are already baked into the handwritten kernels of projects like SGLang, Flash Linear Attention, and DeepGEMM. The agent pulls optimization techniques from across different codebases, different languages, and different hardware platforms, and distills them into “optimization blueprints” that include适用条件、转换方法、resource constraints, and validation evidence. When it encounters a new kernel, it starts from these blueprints, and re-evaluates tiling, memory access, and resource allocation using profiling and layered testing.
On the KDA Decode kernel, the agent noticed the original implementation tiled along the V dimension, which forced the same FP32 normalization and gating calculations to be repeated four times. By merging tiles, storing shared intermediate results in registers, and replacing redundant calculations with a single warp-level reduction, the agent traded a small amount of parallelism to eliminate the root cause of redundant work — delivering a 1.71x speedup over the original implementation.
In this entire process, human engineers only handled defining goals and constraints, and reviewing changes that impacted numerical semantics, concurrency behavior, or production risk.
Figure 4: Performance evolution of a representative KDA Decode kernel from baseline to production-ready version
The Final Result
GLM-5.3-Flash was tested on OpenCode and OpenRouter under the anonymous model name Ox-Alpha. Within one week of going live, it became the most used model on both platforms, processing 62 trillion tokens in just six days.
Z.ai is clear that this isn’t full recursive self-improvement (RSI) yet. Goal selection, problem boundary setting, and risk assessment are still handled by humans. But the numbers — two weeks, 3x throughput, 100,000 accelerators — tell a clear story: the pace at which this line between human and model work shifts won’t slow down just because humans want it to.
Models optimize the system, and the system runs the model. That loop is already spinning.
