Strata: Running Massive AI Models on Hardware You Already Own
Part 1 of 2. How Strata makes large language models practical on heterogeneous, often aging hardware. For the benchmarks and my two builds, see Part 2.
The idea that started it all
In August 2026, I came across a paper that changed how I think about local AI inference: FreeToken.
The idea is simple. Instead of treating a personal computer as a system with an underpowered GPU, treat the entire machine (GPU, CPU, memory, and storage) as the inference platform.
For years, the local AI conversation revolved around one number: how much VRAM do you have? If the model didn’t fit, you bought more hardware, offloaded layers and ate the slowdown, or picked a smaller model. FreeToken demonstrated that a carefully designed runtime can make much larger Mixture-of-Experts models usable on machines that are nowhere near datacenter grade.
That paper convinced me to build my own personal AI lab, and the software that made the idea worthwhile was Strata.
What exactly is Strata?
Strata is an open-source inference engine built on the llama.cpp/ggml ecosystem, specialized in serving large MoE models across mixed hardware. The MoE part matters: these models have a far larger total parameter count than the number of parameters activated for any single token.
The simplest analogy I’ve found is running a video game.
A modern game doesn’t keep every texture and asset in VRAM. Frequently used resources live in VRAM, less common ones in system RAM, and the rest streams from storage, while the CPU and GPU share the work. The analogy isn’t perfect, but the principle is the same: use the whole system instead of treating GPU memory as the only thing that counts.
Why Mixture-of-Experts changes the equation
In a dense transformer, most of the model’s weights run for every generated token. In an MoE model, a router selects a small subset of specialized networks (the experts) for each token. The model gets an enormous total parameter count without activating all of it on every token.
That creates an obvious question: if only a few experts are needed right now, why should every expert be squatting in scarce VRAM?
Strata exploits that opening. It keeps the latency-critical work on the GPU, caches the most commonly used experts there, and executes less common experts on the CPU. VRAM and system RAM become complementary resources instead of RAM being treated as overflow storage.
How Strata divides the work
GPU VRAM: the fast workspace
VRAM holds the latency-sensitive work: attention, routing, shared components, and frequently used experts. Strata’s adaptive GPU expert cache keeps the useful experts close to where they execute quickly, and the working set shifts as the workload changes. More VRAM means better cache residency, but total model size no longer has to be smaller than your VRAM capacity. That’s the big idea.
System RAM and CPU: more than overflow
System RAM holds a much larger pool of experts. On supported systems (roughly 2015 or newer), Strata can even compute an expert on the CPU when it isn’t in the GPU cache. That CPU work overlaps with what the GPU is already doing, rather than stopping generation, copying the expert into VRAM, and only then executing it. Older CPUs can participate, but their instruction support and memory bandwidth will show.
NVMe: selective on-demand access
Storage plays its own role. Some supported architectures include very large lookup structures, like n-gram embedding tables. A lookup isn’t a matrix multiplication: for a given token, the runtime may only need a handful of entries. Strata keeps compatible tables disk-backed and reads the necessary entries on demand. Some configurations can also file-back model data when RAM runs short, though hammering your SSD is no substitute for having enough memory.
NVMe is not magically as fast as VRAM. The point is to avoid touching data you don’t need at all.
But isn’t PCIe and system RAM much slower?
Yes, absolutely. The bandwidth and latency differences are real physics, and Strata can’t eliminate them.
What it can do is shrink the part of inference that has to wait on slow transfers:
- Expert caching keeps frequently activated experts in VRAM.
- Concurrent CPU/GPU execution lets an uncached expert run on the CPU while the GPU handles other work.
- Different strategies for prompt processing and decoding, because those are two workloads with very different performance profiles.
- Selective disk-backed lookups avoid loading giant tables that get touched sparsely.
Some workloads will still bottleneck on CPU compute, RAM bandwidth, PCIe, or SSD access. The software just gives you more ways to navigate those tradeoffs.
Multi-Token Prediction and n-gram tables
Two model-specific details deserve mention, because generic talk of “CPU offloading” undersells what’s happening.
Get 0xtb | Tyrrell’s stories in your inbox
Join Medium for free to get updates from this writer.
Multi-Token Prediction (MTP) proposes multiple future tokens and lets the main model verify them. When proposals are accepted, the system advances more than one token per verification pass. Real gains depend on the model, the acceptance rate, and the configuration. It’s not a magic multiplier, but when it works, it works.
N-gram embedding lookup tables solve a different problem. Instead of pushing a giant table through normal transformer compute, Strata retrieves the specific entries the current input needs. That’s what makes selective storage access viable in a way it never would be for arbitrary dense layers.
Both are model-dependent, so don’t expect them to apply identically to everything Strata supports.
Long context without defaulting to 4-bit KV cache
Long conversations create a second memory problem: the key-value cache grows with the context, and it adds up fast.
Strata documents support for different KV-cache precision and memory-management options, including configurations that keep parts of the cache in system RAM and move what’s needed into VRAM. That makes larger context windows practical on memory-limited GPUs without forcing 4-bit KV quantization as the only option.
The tradeoff is real, though. A model configured for 128K tokens still has to process 128K tokens, and moving cache data costs something. Long-context capability is not the same thing as free long-context performance. Ideally, model builders solve this at the architecture level instead of leaving it to the engine, but that’s a topic for another post.
What about 3-bit quantization?
Quantization compresses weights into lower-precision formats, and for local AI it often decides whether a model fits at all.
Strata supports model-specific GGUF quantization options, including several low-bit formats. Two interest me: IQ3_XXS, which prioritizes a smaller footprint, and IQ3_S, which preserves more fidelity at a higher memory cost. In my testing, IQ3_S lands near some 4-bit quants in quality, which is remarkable when 4-bit is the normal format for most local deployments.
Neither is automatically the right choice. It depends on the model, the hardware, the task, and whether you can actually measure the quality difference in your own work. The project’s supported models documentation is the best starting point for model packs and memory estimates.
I run IQ3_XXS on both of my systems because I value responsiveness and capacity. Part 2 covers the actual measurements, including why one of my machines decodes three and a half times faster than the other despite losing on prompt processing.
Does Strata require modern hardware?
Not necessarily. There are experimental build paths for older x86 CPUs, including Sandy Bridge-era chips that predate AVX2. That tells you where the project’s priorities are, though I wouldn’t expect great performance from an ancient 2011 machine.
There’s no magic minimum for PCIe generation, DDR version, memory channels, or SSD speed, because the limiting factor moves around with the model, the quantization, the context length, how many experts stay resident on the GPU, and the CPU/GPU balance.
And to be clear: a 12GB GPU can participate in configurations for supported large models, but that does not mean any old 12GB GPU and 32GB RAM box will run every 180B model at interactive speed. Total system capacity and execution strategy still matter.
How is it different from llama.cpp, vLLM, or KTransformers?
These projects overlap, but they prioritize different things.
llama.cpp is the broad, portable foundation most local AI users know: GGUF models, quantization, lots of hardware backends. Strata draws on that ecosystem but implements specialized strategies for particular models and heterogeneous execution.
vLLM is oriented toward high-throughput serving: scheduling, batching, efficient inference for concurrent requests. That’s a different optimization target from making an oversized model usable on an eclectic personal workstation.
KTransformers is closest in spirit, since it also explores heterogeneous CPU/GPU inference for large models.
Paper comparisons only get you so far, though. Here’s what I’ve ran so far: llama.cpp, LM Studio, Ninfer, and ollama. The only one that was worth using, to me, was Ninfer, and even that took heavy modification to squeeze as much performance as I could out of my V100S cards, via a fork built for exactly that purpose (ninjer-v100-tpx). It worked well. The catch is that it only supports Qwen 3.8 27B.
Qwen 3.8 27B isn’t a bad model. It’s just that Qwen 3.8 Flash Next is better, and I can run it properly through Strata on the same hardware.
What makes Strata impressive is that it gives people a real way to serve supported large models on hardware they already own or can find secondhand. Last year, nobody thought a 180B parameter model would run on an old 16GB gaming card from 2022. Now people with AMD, NVIDIA, and Intel GPUs are all doing it.
Why this matters
My favorite part of this story isn’t that an old CPU can technically load a huge model for a screenshot. It’s that smarter inference software is changing what real machines are good for.
FreeToken made me think differently about a personal AI lab. Strata made me want to build one.
In Part 2, I’ll walk through my two systems: a Xeon workstation with an RX 6950 XT, and a Ryzen machine with two 32GB V100S cards. I’ll share their prompt-processing and decoding speeds, and how the faster one became the workhorse for my local agentic cybersecurity research. After that, I’ll get into what I want open models themselves to work toward, now that the inference side has caught up.
