Buried in Apple’s documentation for the Siri AI Developer Preview is a sentence any agent developer will recognize: app skills use a subset of the Agent Skills format. That means a folder per skill, a SKILL.md file, YAML frontmatter with a name and a description, and a Markdown body of instructions. It's the same file layout coding agents load today, now compiled into iPhone apps and read by Siri.

That’s the clearest sign of what Apple shipped on November 3, 2026. Starting with the iOS, iPadOS, macOS and visionOS 27.3 betas, any developer can expose custom actions and content to Siri AI, Apple’s new conversational Siri, not just functionality that fits Apple’s predefined app schemas. Apple’s developer news post says App Store submissions open in spring, and that the features reach customers a few months after that.

Read as an agent developer, the preview maps almost one-to-one onto the architecture you’d build for any tool-calling model. This piece walks through that mapping (tools, generated arguments, skills and evals), with the concept in focus being how Siri decides what to call, and what you can and can’t control about it.

The mapping, in one table

• Tool / function definition: @AppIntent(.tool) → ToolIntent. Marks an app intent as something Siri can call.
• Tool and parameter descriptions: title, description, @Parameter(title:description:). What the model reads to pick a tool and fill arguments.
• Structured output for arguments: @AppEntity(.generable) + GenerableEntity. Lets the model build an entity directly from the request.
• Retrieval: Spotlight index, EnumerableEntityQuery. How Siri finds your content to pass into tools.
• Skill / playbook: Skills.skillsbundle/Skills/<name>/SKILL.md. Natural-language guidance for multi-step tasks.
• Evals: SiriEvaluations framework. Scripted multi-turn scenarios with deterministic checks.
• Permission policy: IntentAuthenticationPolicy, IntentVisibility. Which data needs authentication, and what Siri can't see at all.

Tools: @AppIntent(.tool)

App Intents have existed for years as the way apps expose actions to Shortcuts, Spotlight and widgets. The preview adds one trait. Annotate an intent with @AppIntent(.tool) and it conforms to ToolIntent, which Apple's docs define as an app intent Siri can call while handling a request.

Here’s a minimal example for a hypothetical recipe app, written against the preview documentation. It hasn’t been compiled, because it needs the 27.3 beta SDK, and Apple warns the API may change during the preview.

import AppIntents

@AppIntent(.tool)
struct ScaleRecipe: AppIntent {
    static let title: LocalizedStringResource = "Scale Recipe"
    static let description = IntentDescription(
        "Changes a saved recipe's ingredient amounts to serve a different number of people."
    )

    // RecipeEntity (an AppEntity) and RecipeStore are the app's own types.
    @Parameter(title: "Recipe", description: "A recipe the person has saved in the app")
    var recipe: RecipeEntity

    @Parameter(title: "Servings", description: "Number of people to serve, from 1 to 24")
    var servings: Int

    func perform() async throws -> some IntentResult {
        try await RecipeStore.shared.scale(recipe, to: servings)
        return .result()
    }
}

The descriptions aren’t decoration. They’re the prompt. Apple’s ToolIntent docs tell you to write each description for someone who doesn't know your app, and to include string formats, numeric ranges and units. That's exactly the advice you'd give for an OpenAI or Anthropic tool schema, because it serves the same purpose.

Apple is also specific about what shouldn’t be a tool:

• Widget configuration intents, which don’t perform an action
• Intents that are only one step inside a larger shortcut
• Intents with side effects a person needs to start explicitly, such as placing and paying for an order

The last one is a policy decision as much as a technical one. Siri can put things in your cart, but Apple’s guidance is that checkout isn’t something to hand the model.

Two schema rules are easy to trip over. If your intent already adopts one of Apple’s Siri-supported app schemas (for example sendMessage for a messaging app), Siri can already call it, and adding .tool on top is a build error. If it adopts a schema that only Shortcuts supports, you promote it with @AppIntent(.tool, schema: ...).

Arguments the model fills itself: generable entities

Normally, when a tool takes one of your app’s entity types as a parameter, Siri finds a value by running that entity’s query against your app. The preview adds a second path. Mark a transient entity with @AppEntity(.generable) (which conforms it to GenerableEntity and requires TransientAppEntity), and Siri's model can create the entity and fill in its properties straight from what the person said.

Apple’s example is a flight search: a request naming an origin, a destination, a date and a cabin class becomes a populated search entity that’s passed to the find-flight tool, with no lookup in the app. In harness terms, this is structured output for tool arguments. The docs add one constraint worth noting: every property of a generable entity must itself be a generable type.

The other path, finding real content the person already has, runs through Spotlight. Siri can surface entities your app puts in the Spotlight semantic index, and Apple’s news post says this now works for any entity in the index, not only ones that fit an existing schema. If an entity isn’t indexed, the system can also retrieve it from an EnumerableEntityQuery.

When Siri can call your app at all

This is the part that differs most from a desktop agent harness, where every tool is always in scope. Per the developer preview docs, Siri considers your app’s tools without the person naming your app only when:

• your app is in the foreground,
• one of its Live Activities is active, or
• the person is continuing a conversation in the Siri app that already mentioned your app.

When your app is in the background, the person has to name it. Apple’s ride-share example makes the foreground rule concrete: while the ride’s Live Activity is showing, “change my ride pickup to

“ goes to the right app without naming it.

Requests can also cross apps. Siri can find an item in one app, resolve a contact, and send the item through a messaging app. For that to work, the entities that move between apps need a Transferable representation.

Skills: progressive disclosure, Siri edition

Tools tell Siri what it can do. App skills tell it how to combine them. Apple’s docs list what a skill should contain: how tool intents fit together for a task, which tool to prefer when several could apply, what to avoid, domain knowledge specific to your app, and guidance for clarifying conversations.

The format, per Apple’s “Orchestrating complex actions with app skills”:

• One Skills.skillsbundle per app target (Xcode has a "Skills Bundle" template), with a Skills/ folder inside.
• One subfolder per skill, containing only a SKILL.md. No scripts, references or other supporting files.
• Frontmatter with exactly two supported fields. name must be 1–64 lowercase letters, digits and hyphens, and match the folder name. description is required, a single line of up to 1,024 characters.
• A free-form Markdown body of up to 10,000 characters.

Here’s one for the recipe app:

---
name: plan-dinner-party
description: Plans a dinner party from saved recipes, scales them to the guest count, and builds one combined shopping list.
---
Ask how many guests are coming before doing anything else.
Search saved recipes with the find recipes tool; prefer recipes the
person has marked as favorites. Scale each chosen recipe with the
scale recipe tool to the guest count, then use the add to shopping
list tool once per recipe. Never remove items the person already has
on their shopping list. If a recipe has no saved servings count,
ask instead of guessing.

Compared with the open Agent Skills specification, Apple’s subset drops a lot. The spec allows optional scripts/, references/ and assets/ folders, plus license, compatibility, metadata and an experimental allowed-tools field. Siri supports none of them. Apple also adds limits the spec doesn't have: a single-line description and a hard 10,000-character body. A Siri skill can describe, but it can't bring code along.

How Siri uses this is the concept worth understanding. Siri reads only the frontmatter to decide whether a skill applies, and reads the body only after it decides the skill matches. That’s the progressive-disclosure design the Agent Skills spec describes. Every installed skill’s name and description is loaded up front, and the full instructions only when the skill is activated, so lots of skills can be installed cheaply and the model only pays for the ones it uses. In practice, the description line does most of the work. Apple's verification advice is to rephrase test requests and check that the description is specific enough to be picked and broad enough not to miss common phrasings.

Three more rules are easy to miss:

• Skills are guidance, not workflows. Apple says plainly that you can’t use app skills to enforce deterministic behavior, and that Siri doesn’t execute the listed tool intents in order. If a step must happen, it belongs in code, in the intent’s perform(), not in the skill.
• Refer to tools by what they do. The docs recommend “call the get weather tool first” over the Swift type name, and say not to repeat information your intents’ titles and descriptions already provide.
• Names must be unique across the app, its extensions and embedded frameworks. If two skills share a name, the system uses one of them, and which one is undefined.

Permissions: what Siri may say, and what it can’t see

Two controls govern data exposure.

IntentAuthenticationPolicy can be set on a whole entity or on individual properties. Apple's example is a bank-account entity where the balance requires local device authentication before Siri presents it. The docs flag a subtlety: a property-level policy only covers how Siri presents that entity in responses. If an intent returns sensitive data as its result, set the policy on the intent itself.

IntentVisibility.internal hides a property from Siri, Shortcuts and other system features entirely, so only your app and its extensions can read it. Apple's example is a bank routing number.

Evals: SiriEvaluations

The last piece is the one most teams skip with their own agents. The new SiriEvaluations framework (iOS/macOS 27.3+) lets you script multi-turn conversations, the SiriScenario and UserTurn types, and state expectations about each turn:

• Calls: Siri calls (or doesn't call) specific intents
• Finds: Siri surfaces specific entities
• Loads: Siri loads specific app skills
• OnScreen: specific entities are on screen
• SiriResponse: the reply contains specific text

Each check is deterministic, but Siri isn’t, and Apple’s docs say so. They recommend building a broad scenario set with indirect, incomplete and unexpected phrasings, running it more than once, and improving your intents, parameter descriptions and skills until more of the set passes, and passes consistently. That’s the same discipline as evaluating any LLM agent: measure pass rates, not single runs. Xcode also adds a Siri Integration Advisor that reviews your App Intents code and suggests improvements.

What you can do today, and what you can’t yet

• Testing is limited to your own devices. You need a development-signed build on a device that supports the preview, or internal TestFlight testers. External TestFlight testers and App Store users don’t get the Siri AI behavior yet, and the Skills verification guidance says to use a device, not the Simulator.
• Availability has gaps. Apple’s news post says Siri AI needs an Apple Intelligence device set to English, French, Japanese, Korean, Portuguese or Spanish, with Traditional Chinese, Norwegian, Turkish and Vietnamese coming soon. It isn’t currently available in the EU on iOS, iPadOS and watchOS, and isn’t available to users under 13.
• There’s a meter. Server-side Apple Intelligence features, Siri AI included, have daily usage limits that may vary, and Apple says expanded access will cost a fee in the future. It also says terms and usage policies for Siri AI experiences are still to come.
• The API may change. The preview docs repeat that warning on page after page.

The takeaway

What Apple shipped isn’t a new idea. It’s an existing one (tools, structured arguments, skills and evals) adopted at the scale of an operating system, with App Intents as the tool layer and a deliberately constrained subset of an open skill format on top. For developers already building agents, the learning curve is mostly vocabulary: tool descriptions are prompts, the skill description is a router, and evals measure a pass rate. For Apple, the constraints are the product. Siri calls your app only when it's in context, Apple's guidance keeps checkout a human action, sensitive fields can sit behind authentication, and skills can suggest but never script. If those boundaries hold up in practice, every app on the phone becomes a tool for one assistant, and the quality of your descriptions decides whether yours gets called.

Sources: — Apple Developer News, “Siri AI is now open to all developers” (November 3, 2026) — developer.apple.com/news/?id=gdk2umrf — Apple Developer Documentation, “Siri AI Developer Preview” — developer.apple.com/documentation/appintents/siri-ai-developer-preview — Apple Developer Documentation, “Bringing custom actions and content to Siri” — Apple Developer Documentation, “Orchestrating complex actions with app skills” — Apple Developer Documentation, ToolIntent and the Siri Evaluations framework - Agent Skills, "Specification" — agentskills.io/specification

inferd, Locals and Rapid-MLX all shipped notable releases in the last two weeks. They make three different bets about what a Mac running a team of agents needs, and each measures itself against a different rival.

On a MacBook Pro with an M2 Pro and 16 GB, running Qwen3.5–4B, Apple’s mlx-lm server writes 87 tokens per second for four concurrent requests and the same 87 for eight, according to Locals’ README. Add agents and the total doesn’t move. Each agent just gets slower.

That flat line is the problem three projects set out to fix. Between October 29 and November 5, 2026, three Apache 2.0 Mac servers shipped releases that explicitly target agent workloads (many concurrent requests, long prompts that grow every turn, tool calls that have to parse) instead of a single chat window:

• inferd (Incept5): a from-scratch Metal engine for big Macs and hybrid models
• Locals (Generated Computers): an MLX-based server with small-batch kernels and GPU-side tool-call grammars, aimed at 16 GB laptops
• Rapid-MLX 1.0: an MLX-based server that bets on agent-client compatibility and single-stream speed

I wrote a separate deep dive on inferd’s caching internals. This piece is the map: why the plateau happens, what each project does about it, what each one compares itself to, and a script to run the comparison none of them have published.

Why throughput plateaus: the small-batch problem

On a Mac, generating one token for one request means reading essentially the whole model from memory once. The Locals paper calls this a “weight stream.” That’s bandwidth-bound work: the arithmetic per weight read is tiny. Batching is how you escape it. If eight requests decode in the same step, you read the weights once and do eight rows of math, so you get roughly eight tokens for the price of one weight read, until the math becomes the bottleneck.

The catch is the shape of those matrix multiplies. With one request, each layer is a matrix-vector product (M = 1 row). Prefilling a long prompt is a big matrix-matrix product (M in the thousands), which GPUs and their kernels are built for. Agent decoding lives in between, at M = 2 to 64 or so. That’s an awkward size, and kernels tuned for the two ends often handle it poorly. If the kernel for 8 rows costs nearly twice what it costs for 4, throughput stops scaling. That’s the shape of the 87-and-87 plateau.

inferd’s architecture doc shows what fixing this looks like. It routes each matrix multiply to a different kernel by row count, and for 3 to 255 rows it uses its own int8-activation path. Its measured linear-layer time per step on the 27B (M5 Ultra) is 22.5 ms at 5 rows, 38.8 ms at 40 rows and 56.2 ms at 80 rows. That’s 16 times the rows for about 2.5 times the time. Locals describes its own fix the same way: GPU kernels written for a few requests at a time.

All three servers attack that middle band. They differ in everything around it.

Bet 1 — inferd: rebuild the engine for big Macs and hybrid models

inferd (v0.4.1, November 5) is C++, Objective-C++ and Metal, with no MLX runtime in the token path, though it vendors MLX’s kernels for some operations. It’s built around Qwen3.8–27B with a DFlash2 speculative drafter, and its hard problem is caching: Qwen3.8 is a hybrid model whose Gated DeltaNet layers carry a recurrent state that can’t be paged like a KV cache. inferd snapshots that state at chosen points, pages the attention KV in shared 8-bit blocks, parks evicted contexts on the SSD, and exposes an API (/v1/contexts, plus an inferd request field) so an agent framework can pin a shared context and fork agents from it.

Headline numbers (maintainer-reported): on a Mac Studio M5 Ultra with 256 GB, 849 tok/s total for 64 agents on 16k contexts. On a Mac mini M6 with 32 GB, 13.3–14.0 completed agent turns per minute on the 27B, versus 8.8–8.9 for Splash 1.5.

Compares itself against: Splash and TensorFold, two agent-oriented Mac engines, with the same weights and drafter.

Says it loses at: single-stream speed (TensorFold’s fast mode is about 19% faster on the 27B), and a handful of agents starting together on some models.

Hardware: macOS 26 or 27. M5-class GPUs are the main target, M3/M4 are lightly tested, and M1/M2 are untested.

Bet 2 — Locals: small kernels, plus tool calls that can’t fail to parse

Locals (v0.2.2, November 4) runs on top of MLX and aims at the most common local-AI machine: a laptop with 16 GB. Its default model is Qwen3.5–4B, and the project says it works on M1 or newer with macOS 14 or newer.

It does two things. The first is small-batch decode kernels. Its README chart on the M2 Pro shows mlx-lm at 61, 82, 87, 87 and 157 tok/s with 1, 2, 4, 8 and 16 concurrent requests, against Locals at 64, 116, 160, 271 and 341. That’s the 3.1x headline at eight requests, and roughly 1.0x at one.

The second, and more unusual, is grammar-constrained decoding moved onto the GPU, based on a paper by Arip Asadulaev of Generated Computers, which points to Locals as its code: Breaking the Space Barrier and its Application to Language Model Inference (arXiv:2610.09139). Here’s the idea in plain terms. A JSON schema or a tool-call format can be compiled into a graph of allowed states, and checking whether the model’s next token is legal amounts to finding a path through that graph. For a large tool grammar, the machinery behind that check can be big. The paper’s comparison point for one grammar is 824 MB. The paper’s theory section proves a space bound for a family of graph-reachability problems. Its application section shows that the grammar machinery inside LLM decoding has the “exactly one path between any two states” structure the theory needs, and turns each proof into a decoding mechanism. Two practical consequences:

• Forced text is written without running the model. When the grammar allows only one continuation (a closing brace, a fixed key name), the engine emits it directly. This is the same idea as “jump-forward” decoding.
• Grammars get small enough to live on the GPU. The paper reports a 1,106-state tool grammar in 3 MB instead of 824 MB, and a schema-constrained stream that decodes faster than an unconstrained one (180 vs 138 tok/s on a 16 GB laptop). The README rounds this to “about 3 MB instead of up to 1.5 GB” across formats.

For agents, that shows up as tool calls that always parse. The README says an agent can only call tools it offered, arguments arrive in the schema’s types, and a reply cut off by its length limit still closes its call. The paper’s agent benchmark has 16 tool-calling agents (Qwen3.5–2B) finishing 2.55 times sooner than mlx-lm with llguidance. The README lists it as 75 s vs 29 s.

Compares itself against: mlx-lm and, in a separate chart, Ollama, llama.cpp and Magnitude. At eight concurrent requests with 1,000-token prompts on an M2 Pro, it reports a first token in 1.6 s against Ollama’s 41.8 s, measured with SemiAnalysis’s InferenceX client.

Says it loses at: a single request with nothing to reuse (about mlx-lm’s speed), prompt-bound work, formats other than 4-bit affine quantization in groups of 64 (mxfp4 ran at mlx-lm’s speed), and the M5. MLX already has kernels for the M5’s neural accelerators, so the gain there is about 1.5x, not 3x.

Bet 3 — Rapid-MLX 1.0: compatibility and single-stream speed first

Rapid-MLX shipped 1.0.0 on October 29 and 1.0.2 on November 5 (dates from its desktop app’s changelog). It’s the most mature of the three and the broadest. Besides text, the 1.0 releases added persistent terminal agent chats and video generation with synchronized audio. On the agent side, its distinguishing bet is wire compatibility. It serves /v1/chat/completions, OpenAI's /v1/responses (what Codex CLI uses) and Anthropic's /v1/messages (what Claude Code uses), ships 27 tool-call parser modules, and says 12 agent CLIs and 3 Python frameworks are tested against real weights every release. Five of them (Claude Code, Codex CLI, Hermes, Aider and DeepSeek Harness) are Tier-1: per the README, each runs the same real bug-fix task against a local 35B model, and a failure blocks the release.

Under the hood it also does continuous batching and a radix prefix cache with snapshots for DeltaNet-style hybrid models, saved to disk on shutdown. That’s the same caching problem inferd solves, approached from inside MLX.

Headline numbers (vendor-run): its README’s comparison table, run on what it labels a Mac Studio M5 Max with 36 GB using Rapid-MLX 0.16.1 (before 1.0), shows Qwen3.8–27B single-stream decode at 37.9 tok/s versus 25.3 for oMLX and 26.0 for LM Studio. With four streams it’s 82.8 vs 75.3 vs 69.7. The footnote says it’s a single run per engine.

Compares itself against: mlx-lm (“up to 4x faster, 1.5x on a typical task”), oMLX and LM Studio.

Read before installing: anonymous, metadata-only telemetry is on by default (it can be turned off with rapid-mlx telemetry off, RAPID_MLX_TELEMETRY=0 or DO_NOT_TRACK=1). The 1.0 release notes also say the new terminal agent harness's network policy is heuristic and its terminal commands aren't sandboxed.

Side by side

inferd

• Version (date): 0.4.1 (Nov 5)
• Engine: Native Metal, own kernels (some vendored from MLX)
• Built for: Big Macs, dozens of agents on hybrid models
• Default model: Qwen3.8–27B + DFlash2 drafter
• Oldest supported chip: M3/M4 lightly tested, M1/M2 untested
• Cache: Paged 8-bit KV, recurrent-state snapshots, SSD tier, explicit context API
• Tool calls: Strict JSON Schema via constrained decoding (documented subset)
• Benchmarks against: Splash, TensorFold
• Default port: 8300
• Telemetry: — (not mentioned in README)

Locals

• Version (date): 0.2.2 (Nov 4)
• Engine: MLX + own small-batch kernels
• Built for: 16 GB laptops, a handful of agents with tool calls
• Default model: Qwen3.5–4B
• Oldest supported chip: M1
• Cache: Conversations kept between turns; agent system prompts kept on disk across restarts
• Tool calls: Per-model tool-call grammar on the GPU
• Benchmarks against: mlx-lm, Ollama, llama.cpp, Magnitude
• Default port: 8321
• Telemetry: — (README says nothing leaves your computer)

Rapid-MLX

• Version (date): 1.0.2 (Nov 5)
• Engine: MLX
• Built for: Compatibility with existing agent CLIs
• Default model: Varies (pinned model aliases)
• Oldest supported chip: M1
• Cache: Radix prefix cache with hybrid-model snapshots, saved to disk on shutdown
• Tool calls: 27 parser modules
• Benchmarks against: mlx-lm, oMLX, LM Studio
• Default port: 8000 (first free in 8000–8009)
• Telemetry: On by default, opt-out

Notice the “benchmarks against” row. None of the three compares itself to either of the others. Each picked a different baseline, on different Macs, with different models. inferd’s numbers are mostly on M5-class machines and 27B-plus models, Locals’ on an M2 Pro with a 4B model, and Rapid-MLX’s headline table on a 36 GB M5 Max. Each project’s numbers look internally careful, and all three document how they measured. But you can’t read a ranking off them.

Run the comparison yourself

All three speak the OpenAI chat API, so one script covers them. This one fires N streaming requests at once and reports the three numbers that matter for agents: total throughput, how long the slowest agent waited for its first token, and what each agent experienced individually. I tested it against a mock server that saturates past four concurrent streams, and it reproduced the plateau.

"""Sweep concurrency against any OpenAI-compatible local server.
pip install httpx
python agent_bench.py --base http://localhost:8321/v1 --model <name> --levels 1,2,4,8,16
"""
import argparse, asyncio, json, statistics, time
import httpx

PROMPT = ("You are agent {i}. Write a numbered list of 12 concrete checks for reviewing a "
          "pull request that touches a database migration. One line each.")

async def one_request(client, base, model, i, max_tokens):
    body = {"model": model, "stream": True, "max_tokens": max_tokens, "temperature": 0,
            "messages": [{"role": "user", "content": PROMPT.format(i=i)}],
            "stream_options": {"include_usage": True}}
    t0, first, chunks, usage_tokens = time.perf_counter(), None, 0, None
    async with client.stream("POST", f"{base}/chat/completions", json=body) as r:
        r.raise_for_status()
        async for line in r.aiter_lines():
            if not line.startswith("data:"):
                continue
            data = line[5:].strip()
            if data == "[DONE]":
                break
            ev = json.loads(data)
            if ev.get("usage"):
                usage_tokens = ev["usage"].get("completion_tokens")
            for ch in ev.get("choices", []):
                delta = ch.get("delta", {})
                if delta.get("content") or delta.get("reasoning_content"):
                    chunks += 1
                    first = first or time.perf_counter()
    end = time.perf_counter()
    tokens = usage_tokens or chunks          # fall back to counting streamed chunks
    decode = tokens / (end - first) if first and end > first else 0.0
    return {"ttft": (first or end) - t0, "tokens": tokens, "decode_tps": decode}

async def level(base, model, n, max_tokens):
    async with httpx.AsyncClient(timeout=None) as client:
        t0 = time.perf_counter()
        rs = await asyncio.gather(*(one_request(client, base, model, i, max_tokens) for i in range(n)))
        wall = time.perf_counter() - t0
    return (n, sum(r["tokens"] for r in rs) / wall, max(r["ttft"] for r in rs),
            statistics.median(r["decode_tps"] for r in rs))

async def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--base", default="http://localhost:8300/v1")
    ap.add_argument("--model", required=True)
    ap.add_argument("--levels", default="1,2,4,8,16")
    ap.add_argument("--max-tokens", type=int, default=256)
    a = ap.parse_args()
    print(f"{'agents':>6} {'aggregate tok/s':>16} {'slowest TTFT s':>15} {'median per-agent tok/s':>23}")
    for n in map(int, a.levels.split(",")):
        n, agg, ttft, med = await level(a.base, a.model, n, a.max_tokens)
        print(f"{n:>6} {agg:>16.1f} {ttft:>15.2f} {med:>23.1f}")

asyncio.run(main())

Against the mock (4-stream saturation, 64-token replies), the output shows exactly what to look for on real servers. Aggregate throughput stops climbing past 4, and per-agent speed halves with each doubling after that:

agents  aggregate tok/s  slowest TTFT s  median per-agent tok/s
     1             87.3            0.09                    99.0
     2            179.3            0.06                    98.6
     4            357.1            0.07                    98.4
     8            371.2            0.08                    49.3
    16            381.7            0.11                    24.8

For a fair comparison: use the same model and quantization on every server (each project’s docs list Qwen3.5–9B among its supported models, which makes it a reasonable common choice), run one server at a time with the others stopped, run each level more than once, and turn speculative decoding on or off consistently. If a server doesn’t send usage in its stream, the script counts content chunks instead, which under-counts servers that pack several tokens into one chunk.

This short-prompt test doesn’t exercise what inferd and Locals emphasize most, which is long shared contexts and agents resuming after a gap. To test that, repeat the run with a long shared system prompt (8–16k tokens) and send a second turn per agent. That’s where prefix caching and SSD tiers either pay off or don’t.

Which one to try

• A 16 GB MacBook, one coding agent with a few subagents, small models: Locals is aimed squarely at you. Its README documents M1 or newer with macOS 14 or newer, and its own measurements are on exactly that class of machine.
• A Mac Studio or 64 GB+ mini running a 27B-class hybrid model with many agents: inferd’s numbers are on exactly that hardware. Check that your chip and macOS version are on its tested list first.
• You mostly want Claude Code, Codex CLI or Aider to work locally without fiddling: Rapid-MLX’s release-blocking compatibility tests are its real advantage. Turn telemetry off if you don’t want it.

The takeaway

For two years, Mac inference servers competed on one number: single-stream decode speed. All three of these projects say, in their own way, that the number is now secondary. The new contest is the middle band of batch sizes where agent fleets live, tool calls that never fail to parse, and caches that understand hybrid models. The good news is that the work is open and the harnesses are published. The missing piece is someone running all three on the same Mac with the same model. Until that happens, treat every headline here as a claim to test, not a ranking.

Sources: — Locals README and CHANGELOG (v0.2.2, 4 November 2026), Generated Computers — github.com/Generated-Computers/Locals — Arip Asadulaev, Breaking the Space Barrier and its Application to Language Model Inference, arXiv:2610.09139 (October 2026), PDF in the Locals repository — inferd README, OVERVIEW.md and docs/ARCHITECTURE.md (v0.4.1, 5 November 2026), Incept5 — github.com/Incept5-AI/inferd — Rapid-MLX README, release notes v1.0.0 and v1.0.2, and desktop app CHANGELOG — github.com/raullenchai/Rapid-MLX — AlphaSignal, “Generated Computers’ Locals Triples AI Agent Throughput on Apple Silicon” (November 2026) — used for discovery; all figures above are from the primary sources

On a Mac Studio with an M5 Ultra and 256 GB, inferd reports 849 tokens per second of combined output from 64 agents running Qwen3.8–27B at once, each on its own 16k-token context, with the slowest agent’s first token arriving in 3.4–3.7 seconds. The same README says, without hedging, that other Mac engines are faster if you only run one stream.

Both claims are true at once, and that is the point of the project. inferd is a native inference engine written in C++, Objective-C++ and Metal by Incept5, released as v0.4.1 on November 5, 2026 under Apache 2.0. The authors say it was written from scratch over October and November, with no MLX runtime in the token path (it does vendor MLX’s quantized matmul, GEMM and flash-attention kernels under MIT). It is built around one observation: an agent workload doesn’t look like a chat.

A chat is one person waiting on one reply. An agent fleet is a planner, a tester, a reviewer and ten readers all hitting the same model, each with a long prompt that grows by a few hundred tokens per turn, each returning a minute later. The project’s own overview lists four things that follow from that: agents arrive in crowds, agents come back, agents share context, and agents need different budgets. Every major feature in inferd maps to one of those.

This piece walks through how it does that, with one concept in focus: why prefix caching gets hard on hybrid models like Qwen3.8, and what inferd does about it.

The model inferd is built around

inferd’s primary model is Qwen3.8–27B at 4-bit, paired with the DFlash2 block-diffusion drafter for speculative decoding. Qwen3.8–27B is a hybrid. According to inferd’s architecture doc, it has 64 layers, and only every fourth one is full attention (4 KV heads × 256 dims, 24 query heads). The other 48 are Gated DeltaNet (GDN) layers, a linear-attention design that carries a fixed-size recurrent state instead of a growing KV cache.

That split has two consequences that pull in opposite directions.

Concurrency gets cheap. Because only 16 of 64 layers keep a KV cache, the docs put KV at 64 KiB per token in bf16. A pure-attention model with the same shape in all 64 layers would need four times that. Each sequence does pay a fixed recurrent-state cost (3 MiB per GDN layer, 144 MiB per sequence), but that doesn’t grow with context length.

Caching gets hard. This is the subtle part. In a normal transformer, the KV cache for a prefix is just a stack of per-token entries. If two requests share the first 10,000 tokens, they can share those 10,000 tokens’ KV pages, and either one can stop reading at any position. A recurrent state doesn’t work that way. The GDN state at token 10,000 is a single summary of everything before it. You can’t slice it at token 6,000, and you can’t rebuild it at token 6,000 without re-reading the first 6,000 tokens. inferd’s architecture doc names this as the core difficulty: cheap KV memory makes many agents easy to hold, but every cache trick (sharing a prefix, branching a conversation, batching sequences together) now needs special handling for each sequence’s recurrent state. The doc says that’s exactly where the existing Mac engines fall down.

So inferd caches two different kinds of things in two different ways.

Pages for attention, snapshots for recurrence

KV is paged and shared by reference. Attention KV lives in 32-token blocks that are refcounted, so one physical block can belong to several running agents and to the prefix cache at once. A prefix trie indexes full blocks by their tokens. When a follow-up turn, a fork, a retry or a second agent reading the same document comes in, it attends to the same physical pages as the first. The KV cache is 8-bit by default (int8 values plus a float scale per position and KV head), which the docs say roughly doubles capacity: a 140 GiB pool holds about 4.5 million tokens at 8-bit versus 2.3 million at bf16.

Recurrent state is snapshotted at chosen points. Since the GDN state can’t be sliced, the scheduler copies it into a spare “snapshot slot” at specific positions as prompts are prefilled. A request then resumes from the deepest cached node in the trie that carries a snapshot. Picking the points is where most of the engineering is:

• Near the end of each prompt, but deliberately 16 tokens early, rounded down to a block boundary. Chat templates re-render the tail of the previous turn slightly differently on the next turn, so the follow-up diverges a few tokens before the old prompt ended. Snapshotting a little early keeps the snapshot inside the part that actually matches.
• At the end of the system prompt, which the front end now sends as a hint. Before this change (dated November 1 in the docs), agents that shared a system prompt but asked different questions shared zero cached tokens, because the only snapshot sat past the point where they diverged.
• At branch points, when a new prompt’s KV match runs at least 256 tokens past the state it restored from.

Speculative decoding has to respect the snapshot rule too. inferd runs speculative decoding for every agent, with the drafter proposing a block of 4, 5 or 8 tokens that the target model verifies in one pass. With attention, rejected draft tokens are easy: you just don’t commit their KV. With a recurrent layer, running the verify pass would advance the state over tokens that may get rejected. So during verification, inferd doesn’t touch the GDN state. It records the layer inputs to per-layer “tapes”, and once it knows how many drafts were accepted, it replays only the committed rows into the state. A small tuner then picks the block size per regime (number of decoding agents × context bucket) from measured acceptance and step time.

What that costs in memory

Here’s the budget, using only the constants in inferd’s architecture doc. As a sanity check, the script reproduces the doc’s own figure of about 4.5M tokens in a 140 GiB 8-bit pool.

KIB, MIB, GIB = 1024, 1024**2, 1024**3
ATTN_LAYERS = 16                      # 64 layers, every 4th is full attention
KV_HEADS, HEAD_DIM = 4, 256
STATE_PER_SEQ = 144 * MIB             # 3 MiB x 48 GDN layers (docs/ARCHITECTURE.md)
# sanity check against the docs' 64 KiB/token in bf16 (K and V, 2 bytes each)
assert 2 * KV_HEADS * HEAD_DIM * 2 * ATTN_LAYERS == 64 * KIB

def kv_int8_per_token():
    values = 2 * KV_HEADS * HEAD_DIM * 1 * ATTN_LAYERS    # int8 K and V
    scales = 2 * KV_HEADS * 4 * ATTN_LAYERS               # fp32 scale per (position, kv head)
    return values + scales

def budget(agents, ctx, kv_bytes):
    kv, state = agents * ctx * kv_bytes, agents * STATE_PER_SEQ
    return kv / GIB, state / GIB, (kv + state) / GIB

kv8 = kv_int8_per_token()
print(f"8-bit KV: {kv8 / KIB:.1f} KiB/token; 140 GiB pool holds {140 * GIB / kv8 / 1e6:.2f}M tokens")
for agents, ctx in [(16, 16384), (64, 16384), (32, 65536), (128, 16384)]:
    kv, st, tot = budget(agents, ctx, kv8)
    print(f"{agents:>4} agents x {ctx:>6}: KV {kv:5.1f} GiB + state {st:4.1f} GiB = {tot:5.1f} GiB")

Output:

8-bit KV: 32.5 KiB/token; 140 GiB pool holds 4.52M tokens
  16 agents x  16384: KV   8.1 GiB + state  2.2 GiB =  10.4 GiB
  64 agents x  16384: KV  32.5 GiB + state  9.0 GiB =  41.5 GiB
  32 agents x  65536: KV  65.0 GiB + state  4.5 GiB =  69.5 GiB
 128 agents x  16384: KV  65.0 GiB + state 18.0 GiB =  83.0 GiB

That’s on top of the weights (the 4-bit model plus drafter download is about 17 GB, per the README). It explains the hardware story in one table. Sixteen agents at 16k fit comfortably on a 64 GB Mac mini. Sixty-four at 16k, or 32 at 64k, is Mac Studio territory. On small machines, the KV pool, not compute, is what runs out first. That’s why the next feature exists.

When memory runs out: preemption and an SSD tier

Three mechanisms keep a crowded server from falling over.

Admission by memory, not by count. The scheduler admits a waiting request only if the KV blocks it will need (prompt plus 512 generated tokens plus a draft block) fit next to what the running agents will still take. The architecture doc says this replaced admission by sequence count after four 33k-token agents filled the pool on a 32 GB Mac mini M6 and deadlocked at 0.02 tok/s.

Preemption back to the cache. If the pool runs dry mid-step, the newest unplanned agent gives back its blocks and goes to the front of the queue. Its prompt snapshot stays cached, so when it’s readmitted it recomputes only its tail.

An SSD tier. Evicted snapshots, and the KV blocks they need, are written to a content-addressed directory on disk (blocks are named by a hash of every token up to the block’s end, so chains share them). On the 27B with 8-bit KV, the docs put one block at about 1.06 MiB and one recurrent-state snapshot at about 190 MiB. A follow-up turn restored from SSD had a time-to-first-token of 0.24–0.33 s against 7.2 s to re-prefill. Below 256 GB of memory, run.sh turns the tier on automatically (up to 64–96 GiB), and it deliberately survives ./run.sh stop, so agents' contexts come back after a restart.

Fan-out handles the case where many agents start on the same long document at once: only the first one prefills it, and the others wait for its snapshot. The docs report 8 agents on one 32k document going from 188.8 s to 24.6 s.

The numbers, and where it loses

All figures below are inferd’s own, as reported in its README and overview at release 0.4.1, measured by Incept5 with the same weights, drafter and harness for every engine compared.

• Mac Studio M5 Ultra, 256 GB, 27B, 64 agents on 16k contexts: 849 total tok/s. No comparison given.
• Mac Studio M5 Ultra, 27B, 32 agents on 16k: 722 total tok/s. Splash 1.5 levels off near 369.
• Mac Studio M5 Ultra, 32 agents on 64k: the last agent’s first token arrives in 3.3–3.4 s. Splash 1.5 takes 48.9 s.
• Mac Studio M5 Ultra, 8 agents start at once (14k prompts, 6,400 tokens out): all done in 125.6 s. TensorFold 1.2.2 takes 177.6 s.
• Mac Studio M5 Ultra, Qwen3.8-Flash-Next, 64 agents on 16k: 1,331–1,346 total tok/s. No comparison given.
• Mac mini M5 Pro, 64 GB, 27B agent fleet, completed turns/min (16k contexts): 18.2–19.0. Splash 1.5 manages 11.4–12.0.
• Mac mini M6, 32 GB, 27B agent fleet, completed turns/min (8k contexts, back to back): 13.3–13.5. Splash 1.5 manages 8.8–8.9.

The same docs are just as specific about where inferd is worse:

• One fast stream. The overview says TensorFold’s fast mode is about 19% faster on the 27B (275 vs 231 tok/s), and that oMLX and TensorFold are faster on Flash-Next.
• A handful of agents starting together on Flash-Next. TensorFold finishes sooner up to about five agents.
• Small-batch work on the minis. Splash is faster for one agent and for 4–8 agents taking turns on the M5 Pro and M6. One example from the README: the 35B MoE at 16k with 4 agents, 315 vs 237 tok/s on the M5 Pro.
• Models without a drafter. mlx-lm and mlx-vlm are faster for a single agent.

That’s the trade in plain terms. Every design choice above (paging, snapshots, admission, preemption, per-agent speculation) costs a little on the single-stream path and pays back once there’s a queue. If your workload is one person and one chat window, inferd isn’t for you. If it’s a coding agent fanning out subagents, or a research job spawning readers over the same documents, the crossover comes early: in the TensorFold comparison, the README puts it at three agents on the 27B.

On quality, the project reports logits matching the mlx_lm reference (top-1 agreement, cosine ≥ 0.9997), and 100 of 100 strict-JSON replies valid on the first attempt.

Using it: the agent-contexts API

Most engines share prefixes implicitly. inferd also lets the agent framework say what to keep, through an extension to the OpenAI API (docs/CONTEXTS.md):

POST /v1/contexts
{"name": "repo-docs",
 "system": "You review pull requests for this repository.",
 "documents": ["<file 1>", "<file 2>"],
 "persist": "ssd"}

That computes the context once and pins it, so it isn’t evicted and doesn’t count against the admission budget. Agents then fork from it with one extra field on an ordinary chat request:

{"model": "qwen3.8-27b",
 "messages": [{"role": "user", "content": "Review PR #4411 for locking bugs."}],
 "inferd": {"context": "repo-docs", "session": "reviewer-7",
            "effort": "low", "max_context": 32768, "priority": 2}}

• session marks the agent's latest turn so it's evicted last (and spilled to SSD before being dropped). The README says such an agent resumes in 0.2–0.7 s instead of a 15–18 s re-read.
• effort, max_context and priority are per agent, so a cheap search helper and an expensive planner can share one server. Priority decides who gets preempted first.
• "cache": "drop" marks a one-off request that reads the cache but leaves nothing behind.
• GET /v1/cache shows what's cached, pinned, parked and free.

There’s one gotcha the docs call out. Qwen3.x chat templates write the reasoning effort into the system prompt, so a context saved at one effort is a different prefix from the same context at another. Save it at the effort its agents will actually use.

Getting started is four commands from the README: download the release tarball for your macOS version, ./install.sh --model 27b, ./run.sh start --model 27b, and point any OpenAI client at http://localhost:8300/v1. The 27B with its drafter runs from 32 GB.

Read the fine print

• It’s days old. The overview says so directly: a small team, built with AI assistance, tested on the machines it names, and meant for a trusted local user, not a shared network service.
• Hardware coverage is narrow. M5-class and newer GPUs (with neural accelerators) are the main target. The M3/M4 family runs on a separate, lightly tested kernel path, and M1/M2 are untested. If you’re on an older Mac, check docs/MACHINES.md before expecting the numbers above.
• Match the build to the OS. The README warns that a build made for macOS 26 computes wrong logits on macOS 27 at prefill chunks of 256+ rows. Download the file for the macOS you actually run.
• The numbers are the maintainer’s. They’re unusually transparent (raw results are in results/, and the harness is in bench/), but they haven't been independently reproduced. The repo ships examples/agents-at-once, which runs 64 or more agents in about five minutes and checks every answer, so you can test it yourself.

The takeaway

For two years, “fast local inference on a Mac” meant one number: single-stream tokens per second. inferd is a bet that the number that matters next is how many agents a Mac can keep moving, and how quickly the twentieth gets its first token. The hard part isn’t batching. vLLM solved paged KV, continuous batching and prefix caching for data-center attention models, and inferd says openly that it reimplements that design. The hard part is that the open models worth running locally are now hybrids, and a recurrent state can’t be paged. Snapshots, branch points and verify tapes are what it takes to make a Mac Studio behave like a small serving cluster for the models people are actually using. Expect the other Mac engines to grow the same machinery.

Sources: — Incept5, inferd GitHub repository: README.md, OVERVIEW.md, docs/ARCHITECTURE.md, docs/CONTEXTS.md (release v0.4.1, 5 November 2026; repository read 11 November 2026) — github.com/Incept5-AI/inferd — Jian Chen, Yesheng Liang, Zhijian Liu (z-lab), DFlash: Block Diffusion for Flash Speculative Decoding, arXiv:2602.06036 (background on the block-diffusion drafter family DFlash2 belongs to)

A production benchmark on one real classifier, what “max F1” hides, and the request rate at which self-hosting stops paying off

Airbnb fine-tuned a 2-billion-effective-parameter Gemma model for a production content classifier. Its max F1 went from 0.563 to 0.902, on par with the best frontier models Airbnb tested. Median latency was 35 milliseconds. GPT-6.1 Sol and Claude Sonnet 5.5 took about 900 milliseconds at the median and several seconds at p99, against a latency budget of 200.

That result alone would be a familiar “small models are underrated” story. The post by Airbnb engineers Clay Gibson and Adel Elmalaha is more useful than that because it puts real numbers on the boundaries. It shows where prompting stops helping, why a probability beats a yes/no answer, and the request rate below which the frontier API is the cheaper option. It’s also the first production comparison I’ve seen that puts TypeSafe’s Jev decision model next to the generative frontier models.

The post doesn’t give a publication date. Its images are dated November 2026, and its pricing uses October 2026 list rates.

The task: a gate, not a chat

The classifier screens content for violations of Airbnb’s pricing and payment transparency policy. It runs before content is published, so it’s in the critical path. It’s called thousands of times per minute and has 200 ms to return a score. A false positive blocks legitimate content, so precision matters more than in a typical moderation queue.

The setup:

• Evaluation set: 10,000 held-out examples, weighted to correct for how production traffic was sampled. A subset of models was re-checked on 184,000 examples, and their max F1 stayed within 0.01 of the reported numbers.
• Fine-tuned model: Gemma4-E2B, trained on hundreds of thousands of de-identified in-domain examples from users who hadn’t opted out of AI training. Training ran through GLOWC, Airbnb’s internal framework for turning open-weight language and vision-language models into classifiers. GLOWC also provides the shared batch-inference and serving infrastructure.
• Serving: every open-weight model ran on vLLM 0.28.0 on a single NVIDIA RTX Pro 6000. Models up to 12B parameters ran in bfloat16, and larger models used public FP8 or NVFP4 checkpoints.
• Prompt: unless noted, every model got the same concise prompt of about 400 words.
• Proprietary models: called through vendor APIs in low-thinking mode. Airbnb found that more thinking didn’t meaningfully improve classification.

A note on the “2B.” Google’s model card lists Gemma 4 E2B at 2.3 billion effective parameters and 5.1 billion in total. The gap comes from Per-Layer Embeddings, which give each of the model’s 35 layers its own small per-token embedding table. Those tables count toward the file size but not toward the “effective” parameters Google uses to describe the model’s per-token compute. Airbnb describes it as about 2B effective parameters.

The concept: a verdict is a point, a score is a curve

This is the part of the post I’d most want an engineer to take away, because it isn’t really about model size.

Most generative models answer a classification prompt with a verdict: “violation” or “not a violation.” A verdict gives you exactly one precision/recall operating point. With the concise prompt, GPT-6.1 Sol landed at 0.82 precision and 0.96 recall. If the business decides false positives are too costly and wants precision at 0.95, the only lever is rewriting the prompt and re-running the evaluation.

A model that outputs a probability works differently. You choose the threshold. Raise it and precision goes up while recall drops; lower it and the opposite happens. Airbnb’s fine-tuned E2B produces a score, and so does Jev, which TypeSafe designed to return a calibrated decision rather than prose. That’s why Airbnb reports max F1 as its headline metric:

• For models with a score, they sweep the threshold and report the best F1.
• For verdict-only models, they report the F1 of the single verdict.

The team normally prefers PR-AUC for imbalanced data like this, but verdict-only models have no curve to integrate. They say PR-AUC and accuracy rank the models almost identically.

The arithmetic is quick to check. Sol’s single operating point works out to F1 ≈ 0.884, a bit below E2B’s 0.902. The bigger difference is that E2B can be moved along its curve to wherever the policy team wants, and Sol can’t. Here is a small script that computes Sol’s F1 and sweeps a threshold on toy data so you can see the curve:

"""Part A of classifier_math.py: a verdict is one point; a score is a curve."""
import random

def f1(precision, recall):
    return 2 * precision * recall / (precision + recall)

sol_f1 = f1(0.82, 0.96)
print(f"Verdict-only model at P=0.82, R=0.96 -> F1 = {sol_f1:.3f}")

random.seed(7)
data = [(1, random.gauss(0.72, 0.15)) for _ in range(1_000)] + \
       [(0, random.gauss(0.30, 0.15)) for _ in range(9_000)]

def pr_at(threshold):
    tp = sum(1 for y, s in data if y == 1 and s >= threshold)
    fp = sum(1 for y, s in data if y == 0 and s >= threshold)
    fn = sum(1 for y, s in data if y == 1 and s < threshold)
    p = tp / (tp + fp) if tp + fp else 1.0
    r = tp / (tp + fn)
    return p, r

best = (0.0, None)
for t in [0.40, 0.45, 0.50, 0.55, 0.60, 0.65, 0.70]:
    p, r = pr_at(t)
    score = f1(p, r) if p + r else 0.0
    best = max(best, (score, t))
    print(f"threshold {t:.2f}: precision {p:.3f}  recall {r:.3f}  F1 {score:.3f}")
print(f"max F1 = {best[0]:.3f} at threshold {best[1]:.2f}")

On the toy data (10% positives, overlapping score distributions), F1 peaks around 0.81 at threshold 0.60. At 0.70, precision rises to 0.95 and recall falls to 0.59. A scored model lets you choose that trade explicitly; a verdict-only model makes it for you.

The results

Airbnb plotted everything on a quality-versus-latency chart. The post only states exact values for some models, so I’m reporting only those.

Fine-tuned Gemma4-E2B. Max F1 went from 0.563 before fine-tuning to 0.902 after, on par with the best proprietary models. Latency was 35 ms median and about 85 ms p99. The authors say no other model came close on both quality and latency at once.

GPT-6.1 Sol and Claude Sonnet 5.5. Quality was at the top of the chart. Latency was about 900 ms median, with multi-second p99s, well outside the 200 ms budget.

GPT-6 Luna. The fastest proprietary generative model Airbnb tested, but it was still slower and less accurate than the open-weight Gemma4–31B running on one GPU.

Jev. The most accurate off-the-shelf model Airbnb tested under 100 ms. Because it returns a probability, it can be thresholded like the fine-tuned model. The post doesn’t give its F1. For readers of my earlier Jev pieces, this is the first production benchmark I’ve seen where a decision model sits next to the generative frontier models. Here it beat Luna at about the same cost, but it didn’t reach the fine-tuned specialist.

Off-the-shelf open-weight models. Many fit within the latency budget, but their quality plateaued well below both the best proprietary models and the fine-tuned classifier.

One footnote is worth repeating for anyone serving newer architectures. Latency didn’t track parameter count. Airbnb notes that vLLM’s current kernels are less optimized for hybrid linear-attention layers than for standard or sliding-window attention, so two models of similar size can have quite different latencies.

The prompt-engineering ceiling

The obvious objection is that a better prompt would close the gap. Airbnb tested it with a detailed 2,700-word prompt that added worked examples and common false positives.

• Proprietary generative models gained one to three F1 points, which brought them roughly level with the fine-tuned classifier. Their latency stayed far outside the budget.
• Models that already met the latency budget gained only modestly, and a large gap remained.

That split is the core argument. Prompting improves the models that are already too slow and does little for the models fast enough to deploy. The authors open the post with a peanut-butter-and-jelly-sandwich analogy to make the same point: there’s a limit to how much better instructions can do. In their list of when to fine-tune, they put it plainly: hundreds of thousands of labeled examples taught distinctions that the 2,700-word prompt couldn’t.

The cost math, including the break-even point

Airbnb expresses every cost as a multiple of one self-hosted GPU, priced at public cloud rental rates for an RTX Pro 6000. The post doesn’t give the hourly rate. Proprietary costs come from the tokens actually used during evaluation, priced at October 2026 list rates without negotiated discounts.

• Frontier models, concise prompt: about two orders of magnitude more than one GPU at a load of thousands of RPM. The detailed prompt added another 20–40%.
• Lower-cost proprietary options: about 5× the GPU cost, with lower accuracy than the fine-tuned model.
• Luna vs. Jev: similar cost on this task. With the detailed prompt, Jev’s cost quadrupled, because its API has no cached-input discount.
• Break-even: self-hosting becomes cheaper above roughly 30 RPM. Below about 15–30 RPM, Claude Sonnet 5.5 and GPT-6.1 Sol cost less than a dedicated GPU.

The comparison is generous to the APIs. Airbnb assumed a 71% cache-hit rate on cached input, which it says is higher than it actually observed.

The two headline numbers are consistent with each other. A GPU costs the same whether it handles 30 or 3,000 requests per minute, until you need a second one. An API bill grows linearly with traffic. If the two break even at 30 RPM, then at R RPM the API costs R/30 times the GPU:

"""Part B of classifier_math.py: what the break-even point implies."""
G = 1.0                 # one GPU's monthly cost, as the unit
BREAK_EVEN_RPM = 30     # Airbnb: self-hosting wins above ~30 RPM
k = G / BREAK_EVEN_RPM  # API cost per unit of RPM

for rpm in [15, 30, 300, 1_000, 3_000, 6_000]:
    print(f"{rpm:>5} RPM -> API costs {k * rpm:6.1f}x the GPU")

for uplift in (0.20, 0.40):  # the detailed prompt's extra cost
    print(f"+{uplift:.0%} prompt cost -> break-even falls to "
          f"{BREAK_EVEN_RPM / (1 + uplift):.1f} RPM")

15 RPM -> API costs    0.5x the GPU
   30 RPM -> API costs    1.0x the GPU
  300 RPM -> API costs   10.0x the GPU
 1000 RPM -> API costs   33.3x the GPU
 3000 RPM -> API costs  100.0x the GPU
 6000 RPM -> API costs  200.0x the GPU
+20% prompt cost -> break-even falls to 25.0 RPM
+40% prompt cost -> break-even falls to 21.4 RPM

A 100× ratio lands at about 3,000 RPM, which matches Airbnb’s “thousands of requests per minute.” This is a simplified model that ignores engineering time and the cost of fine-tuning, but it shows why the result is so lopsided. A self-hosted GPU is a fixed cost, so once one GPU can carry the traffic, every extra request is essentially free. With an API, every extra request costs money. Better prompts that add tokens make this worse by lowering the break-even point.

When to fine-tune, per Airbnb

The authors give four conditions that should all hold:

1. High volume. Above roughly 30 RPM for this task, one GPU is cheaper than frontier APIs.
2. Tight latency. E2B returned scores at about 85 ms p99, while frontier APIs ran about 900 ms at the median. Faster API tiers exist, but at a premium the post doesn’t quantify.
3. A need for a threshold. Scores let you move the precision/recall trade-off without rewriting prompts.
4. Labeled data. Hundreds of thousands of in-domain examples taught distinctions that a 2,700-word prompt couldn’t.

They’re also explicit about the opposite case. For low-volume or frequently changing tasks, a frontier model with a good prompt can be the better choice. The results come from a single classifier, though the authors say they saw similar trade-offs with other models they fine-tuned.

The takeaway

This is the third company post I’ve covered recently that reaches a similar conclusion, though by different routes. Netflix trained a recommender with a scoring head that never generates text. Pinterest distilled a frontier teacher into a 4B student. Airbnb fine-tuned a 2B-effective model and found that the latency budget, more than the leaderboard, determined which model it could use.

The general lesson is that what you need from the model’s output determines which model fits. A gate that has to answer in 200 ms with an adjustable threshold, thousands of times a minute, needs a score, low latency, and a fixed cost. Frontier models are the right tool for writing the labels, handling low-volume tasks, and covering cases nobody has labeled yet. For high-volume classification in production, Airbnb’s numbers favor a small fine-tuned model running on one GPU.

Appendix: the full script

The two snippets above are excerpts. This is the complete classifier_math.py in one block, so you can copy it into a file and run it with python3 classifier_math.py. It's plain Python with no dependencies.

"""Two pieces of arithmetic behind Airbnb's classifier benchmark.

Part A: why a score you can threshold beats a fixed verdict.
Part B: what Airbnb's break-even and cost-ratio figures imply about each other.

Pure Python, no dependencies. Run: python3 classifier_math.py
"""

import random

def f1(precision, recall):
    return 2 * precision * recall / (precision + recall)

# ---------------------------------------------------------------------------
# Part A: a verdict is one point; a score is a whole curve
# ---------------------------------------------------------------------------
# Airbnb reports GPT-6.1 Sol (concise prompt) at precision 0.82, recall 0.96:
# a single operating point. Its F1 is fixed unless you rewrite the prompt.
sol_f1 = f1(0.82, 0.96)
print(f"Part A: verdict-only model at P=0.82, R=0.96 -> F1 = {sol_f1:.3f}")
print("        (Airbnb's fine-tuned Gemma4-E2B reported max F1 = 0.902)\n")
# A toy scored classifier: 10% positives, scores drawn from overlapping
# distributions. Sweep the threshold and watch precision/recall trade off.
random.seed(7)
data = [(1, random.gauss(0.72, 0.15)) for _ in range(1_000)] + \
       [(0, random.gauss(0.30, 0.15)) for _ in range(9_000)]

def pr_at(threshold):
    tp = sum(1 for y, s in data if y == 1 and s >= threshold)
    fp = sum(1 for y, s in data if y == 0 and s >= threshold)
    fn = sum(1 for y, s in data if y == 1 and s < threshold)
    p = tp / (tp + fp) if tp + fp else 1.0
    r = tp / (tp + fn)
    return p, r

print("  threshold  precision  recall   F1")
best = (0.0, None)
for t in [0.40, 0.45, 0.50, 0.55, 0.60, 0.65, 0.70]:
    p, r = pr_at(t)
    score = f1(p, r) if p + r else 0.0
    best = max(best, (score, t))
    print(f"  {t:9.2f}  {p:9.3f}  {r:6.3f}  {score:.3f}")
print(f"  max F1 on this toy data = {best[0]:.3f} at threshold {best[1]:.2f}")
print("  A precision-first policy can pick a higher threshold instead;")
print("  a verdict-only model has no dial to turn.\n")

# ---------------------------------------------------------------------------
# Part B: break-even arithmetic
# ---------------------------------------------------------------------------
# Model: one self-hosted GPU costs G per month no matter the load (until you
# need a second worker). A pay-per-token API costs k per request, so its
# monthly bill scales linearly with requests per minute (RPM).
# Airbnb: self-hosting wins above ~30 RPM. That pins k: k * 30 = G.
G = 1.0                      # express everything in "GPUs' worth" of cost
BREAK_EVEN_RPM = 30
k = G / BREAK_EVEN_RPM

print("Part B: frontier-API cost as a multiple of one GPU, if break-even is 30 RPM")
for rpm in [15, 30, 300, 1_000, 3_000, 6_000]:
    print(f"  {rpm:>5} RPM -> API costs {k * rpm:6.1f}x the GPU")
print("  ~100x ('two orders of magnitude') lands at ~3,000 RPM,")
print("  matching Airbnb's 'thousands of RPM' description of the workload.\n")

# The detailed 2,700-word prompt added 20-40% to frontier cost.
for uplift in (0.20, 0.40):
    print(f"  +{uplift:.0%} prompt cost -> break-even falls to "
          f"{BREAK_EVEN_RPM / (1 + uplift):.1f} RPM")

Sources: — Clay Gibson and Adel Elmalaha, “Frontier quality at 1% of the cost: a production benchmark of LLM classifiers,” Airbnb Tech Blog (undated; assets and pricing from October–November 2026) — Google, Gemma 4 model card (ai.google.dev): E2B effective and total parameters, layer count, Per-Layer Embeddings — classifier_math.py: my own arithmetic on Airbnb's published figures, given in full in the appendix and run with its output shown above

Stripe says a change to how its Radar fraud model averages training checkpoints improved recall by over 8% in a controlled comparison, which it estimates is worth tens of millions of dollars a year in additional fraud prevention for its merchants. Engineers prioritized 38 ideas, and AI agents did the rest: they implemented the changes, trained the models, debugged failed runs, and evaluated results.

That’s the headline from Stripe’s October 19 engineering post on scaling Radar. But the more instructive part of the post is a failure. Before the agents found that fix, Stripe trained a model on nearly 14 times more data, watched it beat the production model, and then had to roll it back once more fraud reports came in. The reason is a problem every team training on real-world outcomes eventually hits: labels that aren’t finished arriving yet.

A second Stripe post, from October 9, covers a different risk problem with a different tool. It fine-tunes a 4B open-weight language model to score whether a business account is fraudulent, and compares that against prompting. Read together, the two posts say a lot about where agents and language models actually help in production ML, and where humans still decide.

Part 1: Scaling Radar’s transaction model

Radar is Stripe’s fraud prevention product. Its transaction model, called Shield, learns representations of core signals, combines them to predict fraud, and scores payments in milliseconds. The team’s hypothesis was that Shield would do better with fuller payment histories. A card that looks unusual across the whole network might be perfectly normal for one particular business.

The problem was data volume. Shield was trained on roughly 800 million transactions, but that was only a fraction of the available history. To keep training efficient, the pipeline kept all known fraud each week and only a random subset of everything else.

For the first experiment, Stripe removed downsampling and used every eligible payment from a nine-month window. The training set went from about 800 million to 11 billion transactions. After one pass through the data the new model was nearly as good as the existing one, and by the third pass it was better on the initial evaluation.

Each pass took about 24 hours, though, so a three-epoch comparison took about three days. That was far too slow for a team that wanted to test ideas daily. Here’s how they cut it down, with the dataset held fixed at 11 billion examples:

• Baseline: What changed: Initial implementation; Time per epoch: 24 hours
• Streaming: What changed: Data loaders stream from S3 into memory; Time per epoch: 12 hours
• Compute: What changed: torch.compile, running ResNeXt's parallel branches concurrently, better CPU-to-GPU transfers; Time per epoch: 7 hours
• Data movement: What changed: Peak memory cut from 1.3 TB to 500 GB, which allowed more data-loading workers and prefetching; BF16 gradient compression between GPUs; tuned communication bucket size; async CPU-to-GPU transfers; Time per epoch: 4.8 hours

That’s an 80% reduction. Note the last row: cutting peak memory wasn’t the goal in itself. It freed room to run more data-loader workers per GPU, and keeping the GPUs fed turned out to be the bottleneck. The post’s lesson is that data loading, computation, and communication had to be tuned together.

The larger dataset also produced a wider embedding layer. Shield grew to about 30 million parameters, three times its previous size, and Stripe says serving latency didn’t increase in load tests or production scoring.

The delayed-label trap

The bigger model looked better offline and in early production experiments. Then more fraud reports arrived, its measured performance fell below the old model’s, and Stripe rolled it back.

The cause is how fraud labels arrive. When someone uses a stolen card, the cardholder might not notice for weeks. Stripe only learns the payment was fraud when the dispute comes in, long after the payment was scored. Stripe published how slowly those reports accumulate:

• 14: Share of fraud disputes reported so far: 41%
• 30: Share of fraud disputes reported so far: 66%
• 60: Share of fraud disputes reported so far: 86%

So when you build a training set, the most recent weeks contain fraud that is currently labeled as legitimate. Nobody has reported it yet.

This is why downsampling had been quietly protecting the model. The downsampled pool, “everything not known to be fraud,” contained both genuinely legitimate payments and fraud that hadn’t been reported yet. Keeping only a random fraction of that pool also kept only a fraction of the mislabeled fraud. Removing downsampling restored all of it. Stripe says the result was about 20 times more fraud labeled as legitimate, while the count of known fraud stayed the same, since all of it was already kept.

The model learned exactly what the data taught it: that patterns of fraud that gets reported late are legitimate, especially in the most recent weeks. It got worse at catching the fraud that matters most to catch early.

This tension is easy to miss. Recent data helps a fraud model keep up with attackers who change tactics. But recent data has the least mature labels. Stripe’s own framing is that waiting gives you more reliable labels at the cost of learning new behavior later.

Here’s a small script that makes the tradeoff concrete. It uses Stripe’s three published points, assumes 0% reported on day 0, and interpolates linearly between them. The interpolation is my simplification; Stripe only published those three points.

# label_maturity.py — how much eventual fraud is still mislabeled, by payment age
import numpy as np

# Stripe's published cumulative reporting curve, plus day 0
days     = np.array([0, 14, 30, 60])
reported = np.array([0.00, 0.41, 0.66, 0.86])

def unreported(age_days):
    """Share of eventual fraud still labeled 'legitimate' at a given payment age.
    Linear interpolation between published points; beyond day 60 is unknown."""
    return 1 - np.interp(age_days, days, reported)

for age in (7, 14, 30, 45, 60):
    print(f"payment {age:2d} days old: {unreported(age):.0%} of its fraud not yet labeled")

# Average mislabel rate if a training set ends today and covers the last N days
print()
for window in (14, 30, 60):
    ages = np.arange(window)
    print(f"training on the last {window:2d} days: "
          f"{unreported(ages).mean():.0%} of fraud in the window is unlabeled on average")

Output:

payment  7 days old: 80% of its fraud not yet labeled
payment 14 days old: 59% of its fraud not yet labeled
payment 30 days old: 34% of its fraud not yet labeled
payment 45 days old: 24% of its fraud not yet labeled
payment 60 days old: 14% of its fraud not yet labeled

training on the last 14 days: 81% of fraud in the window is unlabeled on average
training on the last 30 days: 63% of fraud in the window is unlabeled on average
training on the last 60 days: 44% of fraud in the window is unlabeled on average

Even at 60 days, by Stripe’s curve, 14% of the fraud that will eventually be reported still hasn’t been. Any team training on outcome labels that arrive late (chargebacks, churn, loan defaults, content-moderation appeals) faces the same curve. The specific numbers differ, but the shape doesn’t.

One more implication: the same problem affects evaluation, not just training. Stripe notes that payments that look legitimate in a recent training or evaluation dataset can pick up fraud labels weeks later. That’s a plausible reason the bigger model looked better at first: a model that misses late-reported fraud scores well against labels that don’t contain that fraud yet. Stripe doesn’t spell out this mechanism, so take it as my reading.

Part 2: Handing agents the experiment lifecycle

Stripe’s answer to slow iteration was what it calls agentic machine learning engineering: a set of tools and instructions that let AI agents carry an experiment from idea to result inside Stripe’s ML infrastructure. According to the post, agents can explore data, prototype features, compute training data, train and evaluate against a baseline, and deploy the feature with an updated model.

A few design choices stand out:

• Features come from one platform. Feature experiments run on Shepherd, Stripe’s feature platform, which the company adapted from the open-source Chronon project. The same feature definition computes historical training data and serves the online equivalent. For agents, this removes a whole class of training/serving mismatch bugs.
• Operational knowledge is packaged for agents. Stripe organizes it as reusable platform procedures plus model-specific docs that agents load per task. That covers datasets, entity keys, timestamps, sampling rules, reproducing features as of a past point in time, and diagnosing failed distributed training jobs. Stripe says this follows the approach from its earlier work on Minions, its one-shot coding agents.
• Agents run jobs, not just code. They query data, inspect workflow executions, read experiment results, write feature code, and run jobs. When a run fails, they inspect the failure, fix the configuration, and retry.
• Humans gate the important points. Engineers review the proposed experiment and its results at defined points, and engineers decide what ships.

The post doesn’t name the models or the agent harness behind this. It also doesn’t say how many experiments per week the team now runs, so the original goal of multiple experiments a week is stated without a measured outcome.

The fix the agents found: averaging from the start

The case study is a real fraud attack during which Shield gave low risk scores to fraudulent payments. Agents got the attack data plus results from earlier work, including the scaling experiments. They used model explanations to find which features pushed scores down on the missed fraud. Humans prioritized 38 ideas, and agents ran them.

The winner came from stochastic weight averaging, or SWA. The technique, introduced by Izmailov and colleagues in 2018, saves copies of a model’s weights at several points during training and averages them into one model, instead of keeping only the final weights. The original paper’s title sums up the claim: averaging weights leads to wider optima and better generalization.

Several agents proposed SWA experiments. Shield already used SWA, but started averaging more than halfway through training: epochs 8 through 13 of a 14-epoch run. Training loss was falling while validation loss rose, which is the classic sign of overfitting, and that gave the agents a reason to test whether including earlier snapshots would help.

The obvious fix didn’t work. Simply stopping training early improved overall ranking but reduced recall in the narrow false-positive range that Radar actually operates in. What did work was combining two changes: averaging from epoch 0 and shortening training to 8 epochs, so the final model averages epochs 0 through 7.

That’s the configuration that raised recall by over 8% in Stripe’s controlled comparison. The dollar estimate comes from Stripe, and the post doesn’t detail how it was calculated.

What’s notable here is that the fix wasn’t exotic. It was a hyperparameter change to a technique the team already used. The value the agents added was throughput: running 38 human-prioritized ideas through implementation, training, debugging, and evaluation, so the one that mattered could surface.

Part 3: Fine-tuning a 4B model to score business accounts

Stripe’s October 9 post tackles a different risk problem: estimating whether a business account is fraudulent, using the same information human reviewers see. Scores are used to rank accounts and set thresholds for routing them to human review.

The setup:

• Input. Card payments, money movement, account activity, and other review features, serialized into a single text record. The serializer turns numerical, categorical, text, and time-series features into one text representation.
• Model. An open-weight 4B-parameter language model. Stripe doesn’t name it.
• Training. Supervised fine-tuning on historical human review labels, where the target response is just the label: 1 for fraud, 0 for not fraud. The loss is negative log-likelihood on that one label token.
• Scoring. At the answer position, take the model’s probability for the “yes” (fraud) token and the “no” token, and renormalize over just those two. The fraud score is P(yes) / (P(yes) + P(no)).

That last step is worth a moment. The model’s probability mass is spread across its whole vocabulary, so P(yes) alone isn’t a clean probability. Renormalizing over the two allowed answers turns it into one. Stripe’s worked example: P(yes) = 0.18 and P(no) = 0.72 give a fraud score of 0.20.

# two_token_score.py — Stripe's label-token fraud score from raw logits
import numpy as np

def fraud_score(logit_yes, logit_no):
    """Softmax over just the two label tokens: e^yes / (e^yes + e^no)."""
    return 1 / (1 + np.exp(logit_no - logit_yes))

# Stripe's worked example, expressed as probabilities over the full vocabulary
p_yes, p_no = 0.18, 0.72
print(f"P(yes) alone:       {p_yes:.2f}")
print(f"renormalized score: {p_yes / (p_yes + p_no):.2f}")

# Same result from logits; any constant shift (the full-vocab normalizer) cancels out
logit_yes, logit_no = np.log(p_yes) + 3.7, np.log(p_no) + 3.7
print(f"from logits:        {fraud_score(logit_yes, logit_no):.2f}")

Output:

P(yes) alone:       0.18
renormalized score: 0.20
from logits:        0.20

Because the score is continuous, it can rank accounts and set routing thresholds, which a single generated “yes” or “no” can’t do.

Results. Stripe measured precision at the recall human reviewers achieve:

• Zero-shot prompting (verbalized probability): Precision at human-review recall: 0.487
• Domain-guided prompting (Stripe’s review criteria in the prompt): Precision at human-review recall: 0.706
• Human review: Precision at human-review recall: 0.790

Stripe doesn’t give the fine-tuned model’s exact precision on this cohort. It reports relative gains instead: about 24% higher average precision than domain-guided prompting, and about 70% higher than zero-shot. Across multiple Stripe risk tasks, it says the fine-tuned models reach more than 21% higher precision at the same recall and consistently beat XGBoost, though no XGBoost numbers are published.

The post is careful about scope. Results come from accounts that were selected for review, so they shouldn’t be read as performance across all Stripe businesses. The labels also incorporate later evidence and outcomes, so evaluation labels aren’t fully independent of what happened afterward.

What fine-tuning changed inside the model. Stripe used linear probes, which are simple classifiers trained on the model’s frozen internal states, to test which concepts became easier to read out after fine-tuning. It probed 44 concepts. The interesting result is about legitimate businesses:

• Risk signals: Base model ROC-AUC: 0.702; Fine-tuned ROC-AUC: 0.729
• Benign signals: Base model ROC-AUC: 0.583; Fine-tuned ROC-AUC: 0.642

Benign signals started near chance (0.5). Stripe’s interpretation is that the base model carried little information about what a legitimate business looks like, and fine-tuning taught it. Eight of the nine benign signals became easier to recover. For example, “consistent business description” went from 0.614 to 0.707, and “established processing history” from 0.644 to 0.712. Stripe notes these are representation changes, not proof of causal mechanisms.

That finding fits the Radar story. In both cases, the hard part of fraud detection isn’t recognizing fraud. It’s recognizing what normal looks like for a specific business, without flagging legitimate but unusual activity.

What the two posts have in common

• Labels are the product. Radar’s rollback came from immature labels. The account model trains on human review labels that fold in later outcomes, and Stripe flags exactly that as an evaluation caveat. In both posts, how labels are made and when they’re trusted shapes the result more than model choice.
• Humans set priorities and thresholds. Engineers chose which 38 ideas to run and what to ship. Human reviewers set the precision benchmark and supply the training labels, and score thresholds decide which accounts reach them.
• The language model isn’t the fraud model. In Radar, agents operate the ML pipeline around a 30M-parameter ResNeXt model that scores payments in milliseconds. The 4B model is used where the input is a whole account history and a human review sits downstream.

The takeaway

The tempting summary is that agents found millions of dollars of fraud. The more accurate one is that agents made a team’s experiment loop fast enough to try 38 things, and that the most important lesson in the post, label maturity, came from a failed experiment rather than a successful one.

If you train on outcomes that arrive late, the practical step is to plot your own reporting curve before your next training-window change, and to re-run your evaluations after labels mature, not just once. The agents can run the experiments, but deciding which labels to trust is still the engineering problem.

Sources:

• Ryan Drapeau and Sai Samant, “Scaling Radar: Fighting fraud with agent-assisted machine learning,” Stripe Dot Dev Blog, Oct 19, 2026 — https://stripe.dev/blog/scaling-radar-fighting-fraud-with-agent-assisted-machine-learning
• Hangzhi Guo, Stathis Vafeias, Florian Hartl, Luvsandondov Lkhamsuren, and Adrian Lam, “Fine-tuning LLMs for risk assessment,” Stripe Dot Dev Blog, Oct 9, 2026 — https://stripe.dev/blog/training-llms-for-risk-assessment
• Stripe, “Shepherd: How Stripe adapted Chronon to scale ML feature development,” Stripe Dot Dev Blog, Apr 15, 2024 — https://stripe.dev/blog/shepherd-how-stripe-adapted-chronon-to-scale-ml-feature-development
• Stripe, “Minions: Stripe’s one-shot, end-to-end coding agents — Part 2,” Stripe Dot Dev Blog, Feb 19, 2026 — https://stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents-part-2
• Izmailov, Podoprikhin, Garipov, Vetrov, and Wilson, “Averaging Weights Leads to Wider Optima and Better Generalization,” arXiv:1803.05407, 2018 — https://arxiv.org/abs/1803.05407
• Xie, Girshick, Dollár, Tu, and He, “Aggregated Residual Transformations for Deep Neural Networks” (ResNeXt), arXiv:1611.05431 — https://arxiv.org/abs/1611.05431
• Alain and Bengio, “Understanding intermediate layers using linear classifier probes,” arXiv:1610.01644, 2016 — https://arxiv.org/abs/1610.01644
• Tian et al., “Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback,” EMNLP 2023 — https://aclanthology.org/2023.emnlp-main.330/

Google’s EmbeddingGemma 2 has 740 million parameters, but if all you’re indexing is text and code, you only load 270 million of them. On MTEB Code, that text-only core scores 78.68, up from 68.76 for the original EmbeddingGemma. Add the 170M vision encoder and the same model embeds photos, screenshots, and video frames. Add the 300M audio encoder and it embeds speech and sound. Every one of those outputs lands in the same vector space, so a typed query can retrieve a voice memo and a photo can retrieve a document.

Google released it on October 6, 2026, under Apache 2.0, with weights on Hugging Face and Kaggle. The technical report followed on arXiv on October 27. This piece covers how the model is put together, what the Matryoshka dimensions actually cost you, how the shared context budget works, and what it takes to run on a Mac. Three short scripts are included. Two run anywhere with Python and NumPy; the third needs the model weights.

The headline is not that a small model got multimodal. It’s that every expensive part of it is optional, and the model card is unusually explicit about which shortcuts hurt quality and by how much.

What “one embedding space” buys you

An embedding model turns an input into a fixed-length vector so that similar inputs end up close together. Search then becomes nearest-neighbor lookup: embed the query, embed the corpus, rank by cosine similarity.

The classic way to search mixed media locally was to chain models. You’d caption images with a vision-language model, transcribe audio with a speech model, and then embed all the resulting text. That works, but every stage adds latency, memory, and a place for information to get lost. A caption can’t describe what it didn’t notice, and a transcript drops everything that isn’t speech.

Google’s developer guide describes EmbeddingGemma 2 as replacing those chains with modular encoders that all project into one shared 768-dimensional space. In practice that means:

• A text query like “ocean waves at sunset” can be compared directly against a photo embedding and an audio embedding, without captioning or transcription in between.
• One input can mix modalities. A product listing with a description, two photos, and a demo clip becomes a single vector.
• Text-only and full-multimodal setups share the space. The developer guide says a query embedded with the 270M text-only configuration can be matched directly against documents embedded with the full model, and that adding an encoder later doesn’t require re-embedding what you already indexed.

That last point is the practical one. You can start with a text-and-code index today and add your photo library next month without rebuilding.

How the 740M parameters break down

The model card splits the parameters into four parts:

• Transformer backbone — Parameters: 130M; Loaded for: Every input
• Embedder — Parameters: 140M; Loaded for: Every input
• Vision encoder — Parameters: 170M; Loaded for: Images, visual documents, video frames
• Audio encoder — Parameters: 300M; Loaded for: Speech and sound
• Total — Parameters: 740M

The backbone plus embedder is the 270M “text model.” With sentence-transformers you choose which encoders to load at startup, and the model card lists four effective sizes: 270M for text only, 440M for text and images, 570M for text and audio, and 740M for everything. The developer guide says disabled encoders are never loaded, so the savings apply to peak memory, not just to the weights on disk.

A detail worth noticing: the embedder is larger than the transformer that does the actual sequence processing. The model card lists a vocabulary of 262,144 tokens and a model dimension of 512. By my arithmetic, a plain token-embedding table at that size is 262,144 × 512 ≈ 134M parameters, which accounts for most of the 140M embedder. In other words, roughly half of the text model is vocabulary. That’s the cost of covering 100+ languages and code with one tokenizer, and it’s the same tokenizer Gemma 4 uses.

The rest of the architecture, per the model card:

• 24 layers, model dimension 512, hidden dimension 2048
• Attention alternates five local (sliding-window, 1,024 tokens) layers for every global one, using grouped-query and multi-query attention
• Mean pooling over the sequence, then a projection from 512 to 768 dimensions

The arXiv report’s abstract credits two design changes for the quality: simplified per-layer embeddings, and a multi-stage training curriculum that progresses through modalities and objectives. The abstract doesn’t break down the individual curriculum stages, so I won’t characterize them further here.

Shared parts with Gemma 4. Google’s launch post says EmbeddingGemma 2 shares Gemma 4’s text tokenizer and audio encoder, so a local retrieval-augmented generation (RAG) pipeline running both models has a lower combined memory footprint than two unrelated models would. Google’s own Mac demo app, Foresight, pairs the two this way: EmbeddingGemma 2 retrieves across images, documents, transcripts, and notes, and Gemma 4 generates the answer, all on device.

One context window, four exchange rates

All modalities share a single 8,192-token context. Each one spends that budget at a fixed rate, which the model card lists:

• Text — Cost: 1 token per subword; Fits in one input (single modality, no text): 8,192 tokens
• Image — Cost: 280 tokens per image (default); Fits in one input (single modality, no text): ~29 images
• Video — Cost: 140 tokens per frame (default); Fits in one input (single modality, no text): ~58 frames
• Audio — Cost: 25 tokens per second; Fits in one input (single modality, no text): ~327 seconds

Three practical consequences follow:

1. Audio is the cheapest per second of content. About five and a half minutes of audio fits in one input. Video at the default 1 frame per second only fits about a minute.
2. The vision budget is adjustable. Images and frames can use anywhere from 70 to 1,120 tokens each. Fewer tokens means faster embedding and more frames per input; more tokens captures finer visual detail. The model card says the lowest budget raises the limit to about 114 images or frames.
3. Interleaved inputs share the budget. A listing with a paragraph, two photos, and a 20-second clip spends the paragraph’s tokens plus 560 for the photos plus 2,800 for the frames.

Video is sampled at 1 frame per second by default, and audio should be supplied as 16 kHz mono.

Here’s a small calculator for planning chunk sizes. It uses only the model card’s default rates and runs anywhere with Python:

# budget.py — how much of each modality fits in EmbeddingGemma 2's 8,192-token context
CONTEXT = 8192
RATES = {
    "image (280 tokens each)": 280,
    "video frame (140 tokens each)": 140,
    "audio second (25 tokens each)": 25,
}

for name, cost in RATES.items():
    print(f"{name:32s} max {CONTEXT // cost}")

# A mixed input: 400 tokens of text, 2 images, a 20-second clip at 1 fps
used = 400 + 2 * 280 + 20 * 140
print(f"\nmixed listing uses {used} of {CONTEXT} tokens ({used / CONTEXT:.0%})")

Output:

image (280 tokens each)          max 29
video frame (140 tokens each)    max 58
audio second (25 tokens each)    max 327

mixed listing uses 3760 of 8192 tokens (46%)

Matryoshka dimensions: what you actually lose

EmbeddingGemma 2 is trained with Matryoshka Representation Learning (MRL). The idea is that training pushes the most important information into the leading dimensions of the vector, so you can keep only the first 512, 256, or 128 values and still get a usable embedding. Shorter vectors mean a smaller index and faster search.

The storage savings are linear. Google’s developer guide gives the example that a million 768-dimensional vectors stored in bfloat16 take roughly 1.5 GB, while the same million at 128 dimensions take about 250 MB, a 6x reduction.

The quality cost is not linear, and it isn’t the same for every modality. Here’s a selection from the model card’s truncation table:

• 768 — MTEB English (text): 68.46; MTEB Code: 78.68; MMEB v2 Overall (image/video/visdoc): 59.01; MSEB Retrieval (audio): 69.54
• 512 — MTEB English (text): 68.41; MTEB Code: 77.24; MMEB v2 Overall (image/video/visdoc): 58.38; MSEB Retrieval (audio): 69.18
• 256 — MTEB English (text): 67.78; MTEB Code: 76.18; MMEB v2 Overall (image/video/visdoc): 56.24; MSEB Retrieval (audio): 66.76
• 128 — MTEB English (text): 65.68; MTEB Code: 71.41; MMEB v2 Overall (image/video/visdoc): 45.65; MSEB Retrieval (audio): 56.71

Read across the 128 row. English text loses under 3 points. The multimodal benchmark loses over 13, and audio retrieval loses nearly 13. The model card’s own guidance follows from this: quality is close to lossless down to 256 dimensions, and 128 is best kept for text-only workloads. The developer guide’s rough rule is that 256 dimensions keeps about 95% of image, video, and speech retrieval quality, while 128 keeps around 90% for text and code but drops to around 75% for the other modalities.

The one mistake that fails silently. A truncated vector is no longer unit length. If your search computes cosine similarity as a plain dot product, which many vector stores do for normalized embeddings, you must L2-normalize again after slicing. The model card warns that skipping this step doesn’t raise an error; it just quietly degrades ranking.

This script shows the mechanics with stand-in vectors. It doesn’t use the model, so it can’t show MRL’s quality behavior, but it does show the normalization trap and the storage arithmetic:

# mrl_demo.py — truncation, re-normalization, and storage math (NumPy only)
import numpy as np

rng = np.random.default_rng(0)

def unit(v):
    return v / np.linalg.norm(v, axis=-1, keepdims=True)

# Stand-ins for unit-length 768-d embeddings
docs = unit(rng.standard_normal((5, 768)))
query = unit(docs[2] + 0.3 * unit(rng.standard_normal(768)))  # closest to doc 2

for dim in (768, 256, 128):
    d, q = docs[:, :dim], query[:dim]
    raw = d @ q                      # dot product on sliced, un-normalized vectors
    fixed = unit(d) @ unit(q)        # true cosine after re-normalizing
    print(f"{dim:4d}d  |q| after slicing = {np.linalg.norm(q):.3f}  "
          f"max dot = {raw.max():.3f}  max cosine = {fixed.max():.3f}")

# Storage for one million vectors in bfloat16 (2 bytes per value)
for dim in (768, 512, 256, 128):
    print(f"{dim:4d}d x 1M vectors: {dim * 2 * 1_000_000 / 1e6:,.0f} MB")

Output:

768d  |q| after slicing = 1.000  max dot = 0.958  max cosine = 0.958
 256d  |q| after slicing = 0.587  max dot = 0.330  max cosine = 0.960
 128d  |q| after slicing = 0.414  max dot = 0.167  max cosine = 0.957
 768d x 1M vectors: 1,536 MB
 512d x 1M vectors: 1,024 MB
 256d x 1M vectors: 512 MB
 128d x 1M vectors: 256 MB

The ranking in this toy happens to survive because all vectors shrink by similar amounts, but the scores are no longer cosines. Any fixed similarity threshold you tuned at 768 dimensions, such as “only return matches above 0.7,” breaks completely. With sentence-transformers you can avoid the issue by passing truncate_dim and normalize_embeddings=True to encode(), which is what the model card recommends. Also note that queries and documents must use the same dimension.

The numbers, and what they’re compared against

Here are the model card’s full-precision results at 768 dimensions:

• Text MTEB multilingual v2 — EmbeddingGemma 2: 61.36; EmbeddingGemma 1: 61.15
• Text MTEB English v2 — EmbeddingGemma 2: 68.46; EmbeddingGemma 1: 69.67*
• Code MTEB Code v1 — EmbeddingGemma 2: 78.68; EmbeddingGemma 1: 68.76
• Image MIEB lite — EmbeddingGemma 2: 65.67; EmbeddingGemma 1: —
• Image MMEB v2 Image — EmbeddingGemma 2: 57.28; EmbeddingGemma 1: —
• Visual docs MMEB v2 VisDoc — EmbeddingGemma 2: 67.84; EmbeddingGemma 1: —
• Video MMEB v2 Video — EmbeddingGemma 2: 50.67; EmbeddingGemma 1: —
• Audio MSEB Retrieval — EmbeddingGemma 2: 69.54; EmbeddingGemma 1: —
• Audio MAEB — EmbeddingGemma 2: 49.87; EmbeddingGemma 1: —

*The EmbeddingGemma 2 model card leaves this cell blank. The 69.67 comes from the original EmbeddingGemma model card on Hugging Face, which reports full-precision 768-dimension scores the same way.

Three things to keep in mind when reading this:

• The comparison column is only the predecessor. The model card compares EmbeddingGemma 2 against EmbeddingGemma 1 and nothing else. Every row below code is new capability, with nothing to compare against on the card.
• Text didn’t improve, and English slipped slightly. Multilingual text is essentially flat (+0.21). English text is down 1.21 points once you pull the predecessor’s number from its own model card. The real text-side gain is code, at +9.92. If your workload is English prose only and you already run the original EmbeddingGemma, upgrading for text quality alone isn’t supported by these numbers. The reasons to upgrade are code, multimodality, the 8K context, and the Apache 2.0 license.
• The strongest claims are Google’s. The launch post calls it best-in-class among sub-1B multimodal embedders on MTEB Code and MAEB, and the arXiv abstract says it beats specialist models and models two to twelve times larger. Those comparisons live in Google’s charts and paper. I haven’t seen an independent leaderboard run yet, so treat them as vendor results until one appears.

Running it on a Mac

Measured speed. Google’s AI Edge team reports that on a MacBook with an M5 Pro, using the GPU and a 70-token vision budget, image embedding takes 37.3 ms per image, or 26.9 images per second. That’s Google’s measurement at the lowest vision budget. Higher budgets will be slower, and I haven’t reproduced it.

Memory. Google’s only published RAM figures are for a Pixel 11 Pro with quantized weights: about 191 MB active for text only and about 567 MB for the full multimodal model. There’s no published Mac figure.

Option 1: sentence-transformers (all modalities). This is Google’s reference path, and the developer guide says it needs sentence-transformers 6.1.0 or later. The model card’s precision rule matters on a Mac: run in bfloat16 or float32, never float16. The model’s activations exceed float16’s range, and it returns NaN or silently degraded embeddings rather than raising an error. Using float32 is the conservative default.

# search_local.py — needs: pip install -U "sentence-transformers[image,audio,video]" transformers
# Replace the file paths with your own files.
import numpy as np
import torch
from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "google/embeddinggemma-2",
    config_kwargs={"audio_config": None},          # text + images only (440M)
    model_kwargs={"torch_dtype": torch.float32},   # never float16
)

photos = ["IMG_0412.jpg", "IMG_0413.jpg", "whiteboard.png"]
photo_embs = np.stack([
    model.encode({"image": p},                     # media takes no task prefix
                 truncate_dim=256, normalize_embeddings=True)
    for p in photos
])

query_emb = model.encode(
    "architecture diagram from the planning meeting",
    prompt_name="SearchQuery",                     # text queries take a task prefix
    truncate_dim=256,
    normalize_embeddings=True,
)

scores = model.similarity(query_emb, photo_embs)[0]
for path, score in sorted(zip(photos, scores.tolist()), key=lambda x: -x[1]):
    print(f"{score:.3f}  {path}")

Two details in that script come straight from the model card. Text inputs should use a task prefix, here via prompt_name="SearchQuery"; skipping it still works but lowers quality. Images, video, and audio take no prefix. If your documents have real titles, the card asks you to format them as title: {title} | text: {content} yourself, since prompt_name="Document" assumes no title.

Option 2: Ollama (text, and images on some tags). Ollama lists five tags: 270m, 440m, 570m, 740m, and latest. Its listing shows image input only on 440m, 740m, and latest, and doesn't list audio input for any tag. The listing also shows a 256K context window, which disagrees with the model card's 8,192 tokens, so I'd trust the model card on that. The text path is one command:

ollama pull embeddinggemma-2:270m
curl http://localhost:11434/api/embed \
  -d '{"model": "embeddinggemma-2:270m", "input": "task: code retrieval | query: retry with exponential backoff"}'

MLX. Google lists MLX among supported runtimes, but its launch post and developer guide link to two different MLX projects, and Google’s MLX integration page only covers Gemma 4 generation. I couldn’t confirm an official MLX embedding command, so I’m leaving it out rather than guessing. For now, sentence-transformers is the verified route for all four modalities on a Mac.

The takeaway

EmbeddingGemma 2’s real contribution is the shape of its tradeoffs, more than any single benchmark row. You pay 270M parameters for text and code, and you add vision or audio only when your data has it. You pick 768, 512, or 256 dimensions for anything multimodal, and keep 128 for text-only shortlists. You trade frames for detail with the vision budget. Each of those choices comes with a published number, so you can decide with data instead of guesswork.

For anyone building local search on a Mac, the practical sequence is to start with the text-only configuration over your code and notes, add the vision encoder for screenshots and photos, and keep everything in one index because the space is shared. The open questions are the independent benchmarks and a first-party MLX path. Until those arrive, sentence-transformers is the reference, and the model card’s float16 and re-normalization warnings are the two mistakes to avoid.

Sources:

• Google, “EmbeddingGemma 2: an open, lightweight multimodal embedding model,” The Keyword, Oct 6, 2026 — https://blog.google/innovation-and-ai/technology/developers-tools/embeddinggemma-2/
• Google DeepMind, EmbeddingGemma 2 model card, Hugging Face — https://huggingface.co/google/embeddinggemma-2
• Google DeepMind, EmbeddingGemma (300M) model card, Hugging Face — https://huggingface.co/google/embeddinggemma-300m
• Grootendorst & Ballantyne, “EmbeddingGemma 2: The Developer Guide,” Google Developers Blog, Oct 6, 2026 — https://developers.googleblog.com/en/embeddinggemma-2-the-developer-guide/
• Schechter Vera, Dua, et al., “EmbeddingGemma 2: Unified and Efficient Multimodal Representations,” arXiv:2610.37364, Oct 27, 2026 — https://arxiv.org/abs/2610.37364
• Google AI Edge, “Google AI Edge with EmbeddingGemma 2,” Google Developers Blog, Oct 2026 — http://developers.googleblog.com/google-ai-edge-with-embeddinggemma-2
• Ollama library, embeddinggemma-2 — https://ollama.com/library/embeddinggemma-2
• Google AI for Developers, Gemma MLX integration page — https://ai.google.dev/gemma/docs/integrations/mlx

A cabinet click, a backsplash search and a countertop save don’t mean much one at a time. Together, they probably mean someone is redoing their kitchen. Pinterest calls that kind of longer-term goal a journey, and in an October 9 engineering post it described how it now infers journeys in production: a fine-tuned 4-billion-parameter LLM reads a person’s chronological activity log and writes back a ranked JSON list of journey names, in that person’s language.

The interesting part isn’t that an LLM can do this. Frontier models already can. It’s what Pinterest had to change to do it for hundreds of millions of people: a distilled small model, a prompt that carries most of the business logic, an output contract that treats “no journey” as a correct answer, and a serving stack that sustains about 775 requests per second on roughly 100 GPUs.

A note on sourcing: Pinterest’s post is on Medium, which blocked automated fetching during my research. Every specific below is drawn from Pinterest’s own text as quoted in search results, cross-checked against ZenML’s independent LLMOps case-study write-up of the same post. Where a detail comes only from ZenML’s summary, I say so.

The pipeline it replaced

Pinterest’s first journey system was a classic multi-stage ML pipeline. ZenML’s summary lists the stages: extract keywords from searches, Pin annotations and the user’s own boards; embed and cluster them; pick representative names; rank and diversify the clusters; and finally classify each journey with heuristics. Pinterest’s post describes that last stage as deciding evergreen versus situational (does engagement persist past a few weeks?) and, for situational journeys, ongoing versus ended.

It worked. Against ordinary interest-based notifications, the legacy journey system lifted email click rate by 88%, push open rate by 32%, and positive survey feedback by 23%. Those are big numbers, and it’s worth being clear they describe the old clustering system versus interest-based notifications, not the new LLM.

But stacking stages compounds errors, and Pinterest names three failure modes:

• Fragmentation. Parts of one project (the tile, the countertop, the shelving) could land in separate clusters and become separate journeys.
• Generic buckets. Clusters could collapse into labels like “Art” or “Shopping.” Pinterest’s verdict on those: “Taxonomically true, useless for personalization.”
• Incidental noise. Clustering has no idea which activity is a goal and which is a passing click, so one stray celebrity Pin could displace a real project.

Each of those is a judgment call that crosses stage boundaries. Is this click part of that project? Is this cluster specific enough to be worth a notification? A pipeline makes those calls locally, one stage at a time. A language model reading the whole log can make them in one pass.

Reframing: journey inference as generation

Per ZenML’s summary, the new system’s input is a semi-static user profile, the user’s board names, and a compact activity log. Pinterest’s post gives the key settings for that log and the decode:

• A 360-day lookback, up to 150 events. Journeys are long-horizon by definition; Pinterest says it tried different windows and event counts to balance cost and relevance.
• Events grouped by (date, action), with a semicolon between Pins and a comma between tags within a Pin. Grouping saves tokens and makes bursts visible: five saves on one day about the same topic look different from one save a month.
• Temperature 0, reasoning tokens off, max 300 output tokens. The goal is a deterministic, replayable read of the same log. Reasoning is off purely to cut online inference cost.

The output is up to about twelve distinct journeys, each 1 to 5 words, in the user’s locale, ranked. The prompt encodes the rules that clustering used to get wrong: don’t merge unrelated topics, drop one-off outliers, treat auto-generated tags as noisy, and only lightly diversify the top of the list among journeys that are already well supported. Searches and saves are treated as the strongest intent signals.

Two design choices stand out.

Empty output is allowed; invention isn’t. Pinterest states this plainly: if the activity doesn’t support a journey, the model may return nothing, but it may not make one up. An empty but valid JSON list counts as a successful inference. They’d rather send nothing journey-specific than a generic “Art.” In evaluation, a fluent, on-brand journey name that isn’t supported by the log counts as a failure, even if a user might have clicked it.

Negative signals live in the prompt. Dislikes stay in the activity log, and Pinterest says this dislike rule replaced a hand-tuned negative-signal filter. That’s a small sentence with a big implication: logic that used to be a separate component with its own thresholds is now a line of instructions. ZenML’s summary notes that Pinterest treats prompt changes as production launches for exactly this reason. The prompt has become a policy surface.

To make the input and output ends concrete, here’s a small toy script. It is not Pinterest’s code; it reconstructs the two contracts the post describes, with invented activity:

"""Toy reconstruction of the two ends of Pinterest's journey-inference contract.
This is NOT Pinterest's code. It illustrates two things their post describes:
  1. Prompt input: events grouped by (date, action), with ';' between Pins
     and ',' between tags within a Pin, capped at a lookback and event count.
  2. Output handling: parse a JSON list, drop non-strings, dedupe
     case-insensitively keeping the first occurrence; an empty list is a
     valid, successful answer.
The sample activity and model output below are invented.
"""

import json
from collections import OrderedDict
from datetime import date, timedelta

LOOKBACK_DAYS = 360
MAX_EVENTS = 150

events = [
    # (date, action, tags of the Pin)
    (date(2026, 10, 2), "search", ["kitchen backsplash"]),
    (date(2026, 10, 2), "save", ["subway tile", "white kitchen"]),
    (date(2026, 10, 2), "save", ["quartz countertop"]),
    (date(2026, 10, 3), "click", ["celebrity red carpet"]),
    (date(2026, 10, 5), "save", ["open shelving", "kitchen"]),
    (date(2026, 10, 5), "dislike", ["gaming setup"]),
    (date(2025, 6, 1), "save", ["old event outside lookback"]),
]

def build_activity_block(events, today):
    cutoff = today - timedelta(days=LOOKBACK_DAYS)
    recent = [e for e in events if e[0] >= cutoff]
    recent = sorted(recent, key=lambda e: e[0])[-MAX_EVENTS:]
    groups = OrderedDict()
    for day, action, tags in recent:
        groups.setdefault((day, action), []).append(",".join(tags))
    return "\n".join(f"{day.isoformat()} {action}: {';'.join(pins)}"
                     for (day, action), pins in groups.items())

def parse_journeys(raw):
    try:
        items = json.loads(raw)
    except json.JSONDecodeError:
        return None            # a failed decode: retry this user
    if not isinstance(items, list):
        return None
    seen, out = set(), []
    for item in items:
        if not isinstance(item, str):
            continue           # drop non-strings
        key = item.strip().lower()
        if key and key not in seen:
            seen.add(key)
            out.append(item.strip())
    return out                 # [] is a valid answer, not an error

print(build_activity_block(events, today=date(2026, 10, 9)))
print()
print(parse_journeys('["Kitchen remodel", "kitchen Remodel", 42, "Open shelving ideas"]'))
print(parse_journeys("[]"))
print(parse_journeys("not json"))

Output:

2026-10-02 search: kitchen backsplash
2026-10-02 save: subway tile,white kitchen;quartz countertop
2026-10-03 click: celebrity red carpet
2026-10-05 save: open shelving,kitchen
2026-10-05 dislike: gaming setup

['Kitchen remodel', 'Open shelving ideas']
[]
None

The distinction in the last three lines is the operational point. A malformed response (None) is a failure to retry; an empty list is a real answer that means "don't send this person a journey notification today." Collapsing those two cases is an easy bug to write, and it would quietly turn "no evidence" into "try again until the model invents something."

Distilling a frontier teacher into a 4B student

Pinterest is direct about why it didn’t just call a frontier model: they can infer coherent journeys without any task-specific training, but they’re too expensive to run regularly for hundreds of millions of people.

So the frontier models became teachers. They generated synthetic training examples, each pairing an activity history with a high-quality target journey list. One detail is worth borrowing: Pinterest found teacher outputs got better when the teacher was allowed extra reasoning tokens before its final structured answer. The student serves with reasoning off; the teacher thinks out loud while making its training data. You pay for reasoning once, at data-generation time, instead of on every production call.

Pinterest fine-tuned open-weight Qwen3 students at 0.6B, 1.7B, 4B and 8B parameters, using supervised fine-tuning with LoRA adapters rather than updating every weight. The 4B model is the production default; ZenML’s summary describes the trade-off as smaller models being faster but weaker on relevance and consistency in hard cases.

The data finding is the most transferable lesson in the post. Quality improved rapidly over the first few thousand examples and then largely stopped. More examples of the same kind added little. What helped was selecting diverse, difficult examples, especially consolidation cases (many signals, one project) and multilingual cases. If you’re distilling a task like this, the budget should go into finding hard examples, not into generating more easy ones.

Serving at notification scale

Pinterest describes the workload as a batch problem with online-shaped latency. There’s a full refresh when the algorithm changes and a daily incremental pass over recently active users.

The stack is NVIDIA Dynamo in front of vLLM decode workers on L40S GPUs. Frontend and decode workers scale independently. The frontend routes with KV-cache awareness, so requests that share prompt structure land on workers already holding that prefix. That matters a lot here because every request shares one long, stable system prompt: the instructions, ranking policy and safety rules are identical for every user, and only the activity block changes. Pinterest also enables prefix caching and chunked prefill and caps concurrent sequences per worker.

The reported numbers: a cluster on the order of 100 GPUs sustains about 775 requests per second end-to-end, with median latency around 1.2 seconds and p95 around 3.0 seconds. Some quick arithmetic (mine, not Pinterest’s): that’s roughly 7–8 requests per second per GPU, and at a sustained 775 per second, about 67 million requests in 24 hours. These are company-reported figures, not an independent benchmark, and nothing in the parts of the post I could verify says how many users a daily pass covers.

The cheaper wins, Pinterest says, are on the batch side:

• Sharding with bounded concurrency. Inference runs as a daily fan-out sharded into small batches, so a slow model response back-pressures one shard instead of stalling the whole pipeline.
• Not recomputing what hasn’t changed. Journeys don’t change hour to hour. Per ZenML’s summary, Pinterest records each user’s last inference time and skips users processed within the previous couple of days. Pinterest makes the point that re-running inference on a journey that hasn’t moved is especially insidious waste: the job completes, the counters go up, and nothing new was learned.

Downstream, each journey name becomes a retrieval query that finds candidate Pins, which existing ranking models then order. Notification logic suppresses journeys the person was recently notified about (fatigue) and journeys they’ve given explicit negative feedback on. On safety, Pinterest says it suppresses inference of sensitive traits at multiple levels before anything is stored; ZenML’s summary adds that both input keywords and generated names pass through content filters.

Did it work?

There’s no ground-truth label for “what someone is really trying to do,” so Pinterest evaluates in layers: side-by-side comparisons of clustering and LLM output for the same users (including misses like over-generalizing), and, per ZenML’s summary, a separate language model that grades sampled outputs on relevance, evidence support, diversity and granularity.

The online result: an initial experiment for US and Canada users, with the clustering system as the control, showed lifts of 1.1% in email click-through and 1.3% in push opens, plus multiple statistically significant sitewide engagement and retention gains. ZenML reports that Pinterest says a later international, multilingual rollout replicated the direction.

Read those two numbers carefully, because they’re easy to misread next to the 88% from earlier. The 88% was the legacy journey system against plain interest-based notifications. The 1.1% and 1.3% are the LLM against that already-strong journey system. A low-single-digit lift on top of a system that already beat its baseline by large margins is a meaningful result at Pinterest’s scale. It is not “the LLM improved clicks by 88%,” which is how a skimmed version of this story could easily get told. ZenML also notes the post gives no population sizes, confidence intervals or absolute baselines for the experiment.

How this differs from Netflix’s approach

If you read my earlier piece on Netflix’s GenRec, the contrast is instructive. Both companies turned a recommendation problem into “feed a language model the user’s history as text.” But they made opposite output choices. Netflix’s model never writes a word: it does prefill-only scoring with a ranking head over catalog item embeddings, so whatever it returns is always a catalog item. Pinterest’s model does generate text, short journey names, and then uses those names as retrieval queries into existing systems.

Generation buys Pinterest something scoring can’t: open-vocabulary, human-readable, multilingual intents that work as queries and as notification hooks. The price is exactly the risk Pinterest spends so much of the post guarding against: a generator can invent. The empty-output rule, the “unsupported name is a failure” evaluation, the output parsing and the content filters are all there because they chose generation.

What’s next

Pinterest lists several experiments in progress:

• Semantic IDs instead of tags. Each Pin gets compressed by a residual-quantized variational autoencoder (RQ-VAE) into five levels of 11 bits, packed into one 64-bit integer and expanded into special tokens when the prompt is built. The bet is fewer input tokens, no dependency on Pin annotations, and a shared item vocabulary between recommenders and LLMs. The cost, per ZenML’s summary, is much harder debugging, since a sequence of semantic IDs isn’t readable the way tags are.
• Offsite signals. Safety-filtered offsite actions such as page visits, add-to-carts, sign-ups and checkouts, added only when onsite activity is sparse, with a capped prompt footprint, and never allowed to rank offsite-only journeys at the top. Pinterest says it’s testing whether better prompting is enough or a fine-tuned offsite-aware model is needed.
• Per ZenML’s summary, also: multi-granularity journeys, generating notification copy jointly, and preference optimization from explicit and implicit feedback.

The takeaway

The pattern here is going to show up everywhere personalization meets LLMs. Use a frontier model to write the training data, with reasoning on. Distill into a small open-weight model that serves with reasoning off. Put the policy in the prompt and treat prompt edits like deploys. Make “nothing” a valid output and “invented” an invalid one. Then spend your efficiency effort on not running the model at all when nothing has changed. A 4B model on L40S GPUs replacing a five-stage pipeline isn’t a story about model capability. It’s a story about turning a chain of fragile heuristics into one contract that can be tested, replayed and reasoned about.

Sources: — Pinterest Engineering, “From Activity to Intent: Generating User Journeys with LLMs” (Lin Zhu, Manan Kalra, Logan Jeon, Ye Liu, et al.), Medium, Oct 9, 2026 (specifics drawn from Pinterest’s text as returned in search results) — ZenML LLMOps Database, “Pinterest: Generative User-Journey Inference at Notification Scale,” 2026 (independent case-study summary of the Pinterest post) — Pinterest Business Community, “Blog: From Activity to Intent: Generating User Journeys with LLMs,” Oct 2026 — daily.dev, “Best of LLM — October 2026” (summary of Netflix GenRec’s prefill-only ranking head served via vLLM)

On October 28, Apple opened orders for the 512 GB Mac Studio with M5 Ultra at $22,299. The next day, Mistral released the open weights of Mistral Large 4, a natively multimodal Mixture-of-Experts model with roughly 1 trillion total parameters. The same day, an mlx-lm pull request appeared listing three community MLX conversions that show what it takes to put that model on Apple Silicon: a 338 GB 2-bit build and a 459 GB 3-bit build that each fit on a single 512 GB Mac, and a 590 GB 4-bit build that needs two.

Those three numbers are the whole story of running frontier-scale MoE models locally. This piece walks through where they come from, what the one published speed figure (18 tokens per second) does and doesn’t tell you, and the practical steps, including the one that trips most people: the code to run it isn’t in a released mlx-lm yet.

What Mistral actually shipped

Mistral calls the model “Le Chonk.” The October 29 release follows a public preview that began October 6; Mistral says the open-weight version got roughly three more weeks of reinforcement learning on top of the preview.

The architecture, per Mistral’s model card and Hugging Face’s launch post:

• ~1T total parameters, 49B active per token (52B counting embeddings and output layers). That’s where the A52B in the repo name comes from.
• 128 routed experts, 4 active per token, plus 1 shared expert that always runs. The first of the 72 decoder layers is dense; the other 71 are MoE.
• Multi-head Latent Attention (MLA) with a DeepSeek Sparse Attention (DSA) indexer, which selects up to 2,048 key positions for each query. Hugging Face’s write-up describes this as two different kinds of sparsity: the router picks which feed-forward experts run, while the indexer picks which earlier tokens get attended to.
• 1M-token context, text and image input (a 1.6B Pixtral vision encoder), text output.
• An embedded DSpark speculative-decoding head: three dense MLA draft layers that propose blocks of three tokens. Mistral reports measuring up to 1.9× higher throughput with it enabled, and notes that gains depend on hardware and workload.
• Official checkpoints in FP8 and NVFP4. The NVFP4 one targets Blackwell GPUs.

The license is the Mistral Ecosystem Open-Weight License, a modified MIT license. It is free to use, modify, fine-tune and commercialize, with one condition: a business offering the model as a service needs a separate commercial license from Mistral.

On benchmarks, Mistral’s own numbers put it at 48% on the Artificial Analysis Coding Agent Index, effectively tied with GLM-5.3 while using 47% fewer tokens per task on average, and 41% on Terminal-Bench 4.0, up 13 points from the preview. (Hugging Face’s launch post gives a different Coding Agent Index figure, 49.8%, so expect the exact number to depend on which reasoning setting and snapshot you look at.) Two caveats are worth keeping. These are Mistral’s reported results, and on the broader Artificial Analysis Intelligence Index v5 the model scores 26, which Mistral itself notes is behind the leading Chinese open-weight models. It’s a strong coding and cybersecurity model, not a uniformly dominant one.

Why a 1T MoE is a memory problem, not a compute problem

Here is the concept that makes every number below make sense.

A Mixture-of-Experts model has two different sizes. Total parameters decide how much memory you need, because every expert has to be resident: the router can pick any 4 of the 128 for the next token, and you don’t know which ahead of time. Active parameters decide how much work each token takes, because only the selected experts (plus the shared expert and the attention layers) actually run.

On a GPU server, the first number is the hard part: you need enough HBM across enough cards to hold a trillion parameters. On Apple Silicon, unified memory flips the economics. A single machine can hold hundreds of gigabytes, and the GPU reads directly from that pool. What limits decode speed is then mostly memory bandwidth: for each generated token, the hardware has to stream the active weights through the GPU.

So two back-of-envelope formulas get you surprisingly far:

• Memory needed ≈ total parameters × bits per weight ÷ 8, plus headroom for the KV cache, activations, and macOS.
• Decode ceiling ≈ memory bandwidth ÷ (active parameters × bits per weight ÷ 8).

The second one is a ceiling, not a prediction. It ignores the KV cache reads (which grow with context), the sparse-attention indexer, routing overhead, and communication between machines. Real systems land well below it. But it tells you which order of magnitude you’re in, and it shows why quantization helps speed as well as fit: fewer bits per weight means fewer bytes moved per token.

The three MLX builds

All three conversions come from the same source: the FP8 checkpoint, run through ml-explore/mlx-lm PR #2018 by contributor jundot. The PR’s description says it loads the native FP8 weights exactly as MXFP8 (the checkpoint uses power-of-two scales over 128×128 blocks, so the mapping is lossless), and that a BF16 copy of the model would be 2.1 TB. Converting with -q dequantizes the FP8 layers and requantizes them at the bit width you ask for. In every build, the router and two indexer projections stay in BF16, as they are in the original checkpoint.

Here’s what the model cards report:

• Text-2bit: 2-bit routed experts, 6-bit everything else, group size 64. 2.58 bits per weight, 338 GB on disk, ~338 GB peak memory (short test), 1 machine.
• Text-3bit: 3-bit affine, group size 64. 3.5 bits per weight, 459 GB on disk, ~459 GB peak memory (short test), 1 machine.
• Text-4bit: 4-bit affine, group size 64. 4.5 bits per weight, 590 GB on disk, ~300 GB peak memory on each Mac (short test), 2 machines (pipeline parallel).

The 2-bit build is the clever one. Since 128 routed experts across 71 layers hold the bulk of a model like this, it pushes only those to 2 bits and keeps attention, the shared expert and the rest at 6 bits. That’s how it lands at an average of 2.58 bits per weight rather than a flat 2.

Three things the table doesn’t show, and that matter:

All three are text-only. Each card states that the vision encoder and the DSpark draft head are not included. If you want image input or speculative decoding, these builds don’t give you either yet.

“Peak memory” comes from a short generation test. Long contexts grow the KV cache. With a 1M-token context window available, a real agentic session can need noticeably more than the card’s figure.

The 4-bit build’s ~300 GB per Mac rules out the cheaper split. TechRepublic points out that two 256 GB Mac Studios cost $21,598, about $700 less than one 512 GB machine. That’s a tempting way to get 512 GB of total memory, but a ~300 GB shard doesn’t fit in 256 GB. For the 4-bit build as published, each Mac needs more than 256 GB, which in the current Mac Studio lineup means the 512 GB configuration. (Whether the 3-bit build could be split across two 256 GB machines isn’t something any of the cards or the PR tested.)

The one speed number, and what it means

The only Mac speed figure published so far for these builds is in the PR’s verification run of the 3-bit model:

• Prompt: 23 tokens at 24.930 tokens/sec
• Generation: 679 tokens at 18.047 tokens/sec
• Peak memory: 459.347 GB

That’s single-stream decode on one machine, and the PR doesn’t say which Mac produced it. One piece of context: the M5 Ultra 512 GB configuration only became orderable on October 28 and, per 9to5Mac, ships in late November to early December. Apple stopped selling the previous 512 GB Mac Studio (M3 Ultra) in March. So the test machine was most likely an older 512 GB Mac rather than a new M5 Ultra, but that’s an inference, not something the PR states.

For a second reference point, Hugging Face’s launch post reports running the llama.cpp Q2_K GGUF on a single M3 Ultra at about 90 tokens/sec prefill and 15 tokens/sec generation. Different quantization and a different engine, so it isn't a like-for-like comparison, but it lands in the same range.

Now the bandwidth math. I wrote a short script that plugs the card numbers into the formulas above, using 52B active parameters (the upper count, including embeddings and output layers) and the 1.2 TB/s memory bandwidth Apple specifies for M5 Ultra:

"""Back-of-envelope sizing for Mistral Large 4 MLX builds on Apple Silicon.

Inputs come from the mlx-community model cards (bits per weight, peak memory)
and Mistral's model card (active parameters). Everything printed is an
estimate: it ignores KV cache, the sparse-attention indexer's reads, and
communication between machines.

"""
ACTIVE_PARAMS = 52e9          # ~52B active incl. embeddings/output (Mistral card)
M5_ULTRA_BW = 1.2e12          # bytes/s, Apple's spec for M5 Ultra

builds = {
    # name: (bits per weight, peak memory per Mac in GB)
    "2-bit (2.58 bpw)": (2.58, 338),
    "3-bit (3.5 bpw)":  (3.5, 459),
    "4-bit (4.5 bpw)":  (4.5, 300),  # ~300 GB peak on EACH of two Macs
}

print(f"{'build':18} {'GB read/token':>14} {'decode ceiling':>16} {'peak/Mac':>9} {'fits 256GB?':>12} {'fits 512GB?':>12}")
for name, (bpw, peak_gb) in builds.items():
    bytes_per_token = ACTIVE_PARAMS * bpw / 8
    # Pipeline parallelism runs the stages one after another for a single
    # stream, so the per-token weight traffic is the same as on one machine.
    ceiling = M5_ULTRA_BW / bytes_per_token
    print(f"{name:18} {bytes_per_token/1e9:14.1f} {ceiling:13.0f} t/s {peak_gb:7d}GB "
          f"{'yes' if peak_gb < 256 else 'no':>12} {'yes' if peak_gb < 512 else 'no':>12}")

# Wired-limit helper: mlx-lm's README says N should exceed the model size in MB
# but stay below the machine's memory. Example for the 3-bit build on 512 GB.

model_mb = 459e9 / 2**20
ram_mb = 512 * 1024

print(f"\n3-bit build is ~{model_mb:,.0f} MB; a 512 GB Mac has {ram_mb:,} MB.")
print("Example: sudo sysctl iogpu.wired_limit_mb=491520  # 480 GB, leaves ~32 GB for macOS")

Output:

build               GB read/token   decode ceiling  peak/Mac  fits 256GB?  fits 512GB?
2-bit (2.58 bpw)             16.8            72 t/s     338GB           no          yes
3-bit (3.5 bpw)              22.8            53 t/s     459GB           no          yes
4-bit (4.5 bpw)              29.2            41 t/s     300GB           no          yes

3-bit build is ~437,737 MB; a 512 GB Mac has 524,288 MB.
Example: sudo sysctl iogpu.wired_limit_mb=491520  # 480 GB, leaves ~32 GB for macOS

Read the “decode ceiling” column as an upper bound for an M5 Ultra, not a forecast. It leaves out the KV cache, the DSA indexer, and expert-routing overhead, all of which eat into it. The measured 18 tokens/sec for the 3-bit build sits at about a third of the 53 tokens/sec ceiling, and on what was likely slower hardware. That gap is normal for MoE decode at batch size 1.

The script also encodes a point that’s easy to get wrong about the 4-bit build: pipeline parallelism doesn’t make a single conversation faster. It splits the layers between the two Macs, so each token passes through Mac 1’s layers and then Mac 2’s. The bytes moved per token are the same as on one big machine, plus a hop over Thunderbolt. Two machines buy you the memory to hold 4-bit weights, not extra decode speed. (Tensor parallelism, which splits each layer across machines so they work at the same time, is what can raise single-stream speed. I covered that trade-off in detail in the earlier piece on Muse Spark Open across two Mac Studios.)

How to actually run it

Step 1: get an mlx-lm that knows the architecture. This is the step most guides will skip. As of November 1, PR #2018 is still open and awaiting review, so pip install -U mlx-lm gets you the latest release (0.32.2, published October 16), which doesn't include the mistral_large4 model code. The model cards say they were tested with mlx-lm 0.32.3.dev13 and MLX 0.32.4. Until the PR merges and ships, install from the PR's branch:

pip install -U mlx
pip install git+https://github.com/jundot/mlx-lm.git@mistral-large4

Once a release includes it, a plain pip install -U mlx-lm will be enough.

Step 2: raise the wired-memory limit (single-Mac builds). The mlx-lm README notes that models large relative to the machine’s RAM can be slow, and suggests raising the system wired limit with sudo sysctl iogpu.wired_limit_mb=N, where N is larger than the model in megabytes but smaller than total memory. For the 459 GB build on a 512 GB Mac, that's a value between about 438,000 and 524,288. The script's example, 491,520 (480 GB), leaves roughly 32 GB for macOS. Treat that as a starting point, not a tuned value.

Step 3: generate. For the 3-bit build on one machine, the PR’s own verification command is:

mlx_lm.generate --model mlx-community/Mistral-Large-4-1T-A52B-Text-3bit \
    -p "The secret to baking a good cake is" -m 1024

Step 4 (4-bit only): two Macs, pipeline parallel. The 4-bit card shows running mlx-lm’s sharded_generate.py example over MLX's ring backend:

mlx.launch --backend ring --hostfile hosts.json \
    sharded_generate.py --model mlx-community/Mistral-Large-4-1T-A52B-Text-4bit --pipeline --prompt "hello"

The MLX distributed docs cover the hostfile. Two notes from them: the ring backend runs over TCP and is mainly meant for Thunderbolt rings, while the lower-latency JACCL backend (RDMA over Thunderbolt 5, macOS 26.2 or later) has to be enabled from macOS Recovery with rdma_ctl enable. The docs describe JACCL as necessary for things like tensor parallelism; the 4-bit card's example uses ring with --pipeline.

One gotcha from the PR: with pipelining, a 2,048-token prefill chunk can trip the macOS GPU watchdog at around 7k tokens. Passing --prefill-step-size 512 avoids it.

What this costs, and who it’s for

A single 512 GB Mac Studio with M5 Ultra costs $22,299 with 1 TB of storage, and the 512 GB option alone is a $15,500 upgrade over the 96 GB base configuration. The 4-bit build doubles that. For comparison, Mistral’s API price is $0.70 per million input tokens and $2.10 per million output tokens.

At those prices, almost nobody runs Le Chonk locally to save money. The case is the one Mistral leans on in its launch: control. The cybersecurity framing is deliberate (Mistral reports 51% on the Artificial Analysis Cyber Index), and security teams are exactly the audience that may not want vulnerability reports or crash logs leaving the building. A Mac that holds the whole model in one box, at roughly 15 to 18 tokens per second based on the two published Apple Silicon tests, is a slow but fully private option for that kind of work.

The takeaway

Mistral Large 4 makes the shape of local frontier inference concrete. Total parameters set the memory bill, which is why the build you can run is decided by quantization: 338 GB at an average 2.58 bits, 459 GB at 3.5, 590 GB across two machines at 4.5. Active parameters set the speed ceiling, which is why a trillion-parameter model can still decode at tolerable speeds when only 52B parameters move per token. And the community tooling moved fast: a working conversion, three builds and a verification run were all public on release day. The only thing still missing is the merge that turns “install from a PR branch” into pip install -U mlx-lm. Watch PR #2018.

Sources: — Mistral AI, “Mistral Large 4 (Le Chonk) is now open weight” (mistral.ai/news/mistral-large-4-oss), Oct 29, 2026 — Mistral AI, model card for mistralai/Mistral-Large-4–1T-A52B (Hugging Face), Oct 2026 — Hugging Face, “Welcome Le Chonk: Mistral Large 4 on Hugging Face,” Oct 29, 2026 — ml-explore/mlx-lm Pull Request #2018, “Add Mistral Large 4 text model” by jundot, opened Oct 29, 2026 — mlx-community model cards: Mistral-Large-4–1T-A52B-Text-2bit, -Text-3bit, -Text-4bit (Hugging Face), Oct 2026 — MLX documentation, “Distributed Communication” (MLX 0.32.4) — mlx-lm README / PyPI project page (wired-memory guidance; release history), Oct 2026–9to5Mac, “Apple reveals pricing for Mac Studio with 512GB of RAM,” Oct 28, 2026 — TechRepublic, “Apple’s 512GB Mac Studio Now Costs $22,299,” Oct 30, 2026 — MacRumors, “Mac Studio With M5 Ultra Chip and 512GB of RAM Launching in October,” Aug 25, 2026

Netflix’s GenRec beat a production ranker that had been tuned for years, and it did so with about 40× fewer Phase-2 labeled training examples. The model is a decoder-only LLM, the same kind of architecture that powers chatbots. In production, though, it never generates any text.

The details are in a paper Netflix posted to arXiv in August 2026 (2608.10257, by Ying Li, Rein Houthooft, Shradha Sehgal, Yaochen Zhu, Arjun Rao and Ashish Rastogi), alongside a Netflix TechBlog post. They describe a recommender that drops thousands of hand-built features in favor of plain-language user histories. The interesting engineering is everything Netflix did to keep that LLM cheap enough to run on real traffic.

This piece walks through the mechanism: what goes into the prompt, what replaces the language-model output, and why “prefill-only” is what makes the design affordable.

Why replace a ranker that already works?

According to the paper, Netflix’s current production rankers depend on a very large set of hand-crafted features covering users, items and interactions, along with higher-order feature interactions and architectures specialized for different recommendation needs. The system works, but it’s expensive to change. Adding a new content type or a new surface can mean new feature engineering, a new model architecture, infrastructure work and new experiments.

An off-the-shelf LLM doesn’t solve this. The paper lists four ways out-of-the-box LLMs fall short as production recommenders:

• They over-recommend globally popular titles.
• They hallucinate titles that aren’t in the catalog.
• They ignore nuanced business constraints.
• They offer limited personalization.

GenRec is built to fix each of those.

Step 1: The prompt is the new feature vector

GenRec verbalizes a member’s history, the item metadata and the request context as natural or lightly structured text. Training data becomes a “conversation.” The user message holds the context (surface, time, device, locale), the profile and history, item details, and a task such as “predict the next titles this member will play.” The assistant message holds what the member actually did next: played, abandoned, thumbed up, and so on.

A long viewing history can easily blow past the token budget, so Netflix curates what goes in. The paper describes four rules:

• Retain in full: high-signal events such as long plays and thumbs-up, kept with rich metadata.
• Omit: recent, low-signal noise such as very short plays and stray clicks.
• Compress: repetitive behavior. A binge session becomes one line instead of twelve.
• Elaborate selectively: extra metadata for new releases and cold-start titles that the base model knows little about.

The authors call this a shift from feature engineering to context engineering, and they put it more bluntly too: “The ‘prompt’ becomes the new feature vector.” The modeling work no longer goes into designing feature interactions. It goes into deciding which events to include, how far back to look, and how to word them inside a fixed budget.

The budget has a direct effect on cost. Netflix swept the number of history events in the prompt and found an elbow, a point past which extra tokens barely helped. By compacting the verbalization, it cut context from about 5,000 tokens to about 1,700, roughly a third, with negligible loss in offline ranking quality. GenRec is largely compute-bound, with serving cost roughly proportional to context length, so serving cost dropped to roughly a third as well.

Step 2: Swap the vocabulary for the catalog

This is the central idea. A normal LLM ends in a language-model head that scores every token in its vocabulary. GenRec adds a catalog-aware ranking head that scores every title.

The paper describes three stages:

1. A verbalizer turns the history, the item metadata and the context into one text sequence.
2. The LLM encodes that sequence, and GenRec takes the hidden state at a pooling position as a single vector h that summarizes the member and the moment.
3. A scoring head combines h with a learned embedding e_i for each catalog item. A softmax over the catalog turns those scores into a probability distribution, which gives the ranking.

The output space is the catalog, so GenRec can’t recommend a title Netflix doesn’t carry. Hallucination is ruled out by the structure of the model, not filtered out afterward. When the catalog is too large to score exhaustively, the paper notes that sampled softmax can be used instead.

Here’s a toy NumPy version of that head. It’s my illustration of the mechanism the paper describes, not Netflix’s code: a pooled vector, a table of item embeddings, one matrix multiply, and a ranking.

import numpy as np

rng = np.random.default_rng(0)

d = 64              # hidden size of the (toy) LLM backbone
catalog_size = 5000 # number of titles the head can score

# Stand-in for the LLM's pooled hidden state for one request.
h = rng.normal(size=d)

# Learned item embeddings: one row per catalog title.
item_embeddings = rng.normal(size=(catalog_size, d)) / np.sqrt(d)

# Catalog-aware scoring: one matrix-vector product scores every title.
scores = item_embeddings @ h

# Softmax over the catalog -> a probability for each title.
probs = np.exp(scores - scores.max())
probs /= probs.sum()

top10 = np.argsort(-scores)[:10]
print("Top 10 item ids:", top10.tolist())
print(f"Probability mass on top 10: {probs[top10].sum():.3f}")
print("Every id is in the catalog:", bool(np.all(top10 < catalog_size)))

Every candidate gets a score from a single vector, so a five-thousand-title ranking costs one forward pass plus one matrix multiply. That leads to the serving decision.

Step 3: Prefill-only serving

LLM inference has two phases. Prefill reads the whole prompt in one parallel pass. Decode then generates output one token at a time, with each new token requiring another step through the model. Most LLM-based recommenders produce item IDs by decoding, often with beam search, which multiplies the steps further. The paper points out that this latency overhead can be prohibitive for large candidate sets.

GenRec skips decode entirely. The paper says Netflix deliberately serves it in a prefill-only configuration: the model reads the context once and produces scores for the full candidate set in a single forward pass. It runs on Netflix’s internal LLM serving stack using vLLM. A separate Netflix TechBlog post describes that stack: vLLM running inside NVIDIA Triton Inference Server. Coverage of that post reports the platform originally ran on TensorRT-LLM before Netflix chose vLLM.

The paper doesn’t say which vLLM interface GenRec uses. If you want to try the same pattern with open-source vLLM, the closest public feature is pooling models. Run a model with --runner pooling and vLLM returns a pooled hidden state instead of generated text. For GPT-style causal models the default is LAST pooling, the final token's hidden state. A custom head on top of that vector gets you most of the way to GenRec's design.

The paper sums up the cost model in one line: inference cost is roughly proportional to model size times context length. Netflix pulls on both levers. It explores smaller and distilled backbones, and it compacts the context. Prefill-only removes the third cost, decode steps, altogether.

If you read the earlier piece on Cloudflare’s Clef, this is a close cousin of that design. Put a structured scoring head on an LLM backbone and score everything in one pass instead of asking the model to write its answer out. One difference: GenRec trains the backbone, the head and the item embeddings jointly, rather than keeping the LLM frozen.

Step 4: Teaching it what Netflix actually wants

Raw viewing logs reward the wrong things if you train on them naively. The paper’s examples: the model could over-recommend binge-watching over discovery, favor video over games, or chase immediate clicks at the expense of exploration and long-term retention.

GenRec’s training loss is a weighted sum of a ranking objective (cross-entropy over the catalog), a language-modeling objective, and other objectives where applicable. The language objective isn’t used at inference. It’s there to preserve the backbone’s text understanding and its responsiveness to steering through the prompt.

Alignment comes from a reward-weighted ranking loss. Separate reward models score each training example on two things: proxies for long-term satisfaction (did this engagement predict that the member came back, or explored more of the catalog?) and rebalancing across content types and launch stages. That score becomes a per-example weight on the loss. High-value engagements count more and low-value ones count less.

Netflix did try full RL. In preliminary experiments, GRPO-style methods gave additional gains over supervised fine-tuning, but the training overhead was too high for now, so the simpler weighted loss shipped. Teams under real cost constraints will recognize that trade-off.

The numbers, and how to read them

All figures below are from the paper.

• Offline: about +1.6% relative MRR over the production ranker, with roughly 40× fewer Phase-2 labeled examples.
• Online: an A/B test on about 10% of Netflix traffic for 4 weeks, on batch-compute surfaces. GenRec showed statistically significant gains on short- and long-term metrics, including +0.006% relative on the core online metric.
• Phase 1 (Netflix-adapted foundation model): +10–20% offline MRR versus starting from an off-the-shelf open-source LLM.
• Phase 2 (ranking post-training): +35–50% offline MRR over a freshly trained Phase-1 model, rising to about 80% after two weeks as the Phase-1 backbone goes stale.
• Scaling: tested at roughly 1B and 10B parameters. MRR improved monotonically as Phase-2 data grew from 1× to 20×, and the larger backbone won under a fixed training budget.

Two caveats are worth keeping next to those numbers.

First, +0.006% is tiny. The paper says it is statistically meaningful at Netflix’s scale, and that’s plausible across hundreds of millions of members. But the headline result isn’t that the LLM is dramatically better. It’s that the LLM reached parity and a little more while using a fraction of the labels and input signals, and that its offline metrics kept improving as the authors added data. The case rests on efficiency and the room left to scale.

Second, the scope is narrow. The test covered batch-compute surfaces, not everything Netflix shows you. The authors call GenRec a first step and say the approach looks promising in at least some scenarios. They don’t claim to have replaced the recommender.

What to take from this if you don’t run Netflix

You probably don’t have hundreds of millions of members, but three of these ideas carry over to smaller systems:

• Budget your context like a cost line. Netflix’s 5,000-to-1,700 cut came from measuring where quality flattened out and deleting everything past that point. Any LLM feature with a long prompt can run the same sweep.
• When the answer is “pick from a known set,” don’t generate it. A pooled hidden state plus a scoring head over your item set makes out-of-set answers impossible and needs no decode loop. That works for routing, tagging, classification and ranking.
• Separate the slow backbone from the fast adapter. Netflix’s Phase-1 model is updated relatively infrequently, while Phase-2 is refreshed often. That’s why label efficiency in Phase 2 matters so much: it lowers the cost of keeping the ranker current every time it’s retrained.

The takeaway

The paper frames this as two shifts: from feature engineering to context engineering, and from bespoke architectures to shared foundation backbones that inherit LLM-style scaling behavior. GenRec doesn’t finish either shift, but it shows a credible path. Keep the LLM’s understanding, discard its text generation, and treat every token in the prompt as cost.

The recommender of the future may well be a language model. In this design, it still won’t need to write anything.

Sources:

• Li, Houthooft, Sehgal, Zhu, Rao, Rastogi, “GenRec: An LLM-Backed Recommendation Ranker at Netflix,” arXiv:2608.10257 (August 2026). This is the source for all architecture details and figures above.
• Netflix Technology Blog, “GenRec: Towards LLM-Native Recommendation at Netflix” (July 31, 2026).
• Netflix Technology Blog, “In-House LLM Serving at Netflix” (2026), cited in the GenRec paper. Details on it come via InfoQ, “Netflix Details Its In-House LLM Serving Platform with Triton and vLLM” (July 2026), and the ZenML LLMOps Database summary, “In-House LLM Serving Infrastructure at Scale” (2026), the source for the TensorRT-LLM history.
• vLLM documentation, “Pooling Models” (accessed October 22, 2026).

Mods put your own code between the model and the permission system. That’s powerful, and it moves the trust boundary.

A Claude Code mod can approve a tool call that one of your ask rules would have stopped to ask you about. It can rewrite a shell command after Claude proposes it and before it runs. It can restyle much of the interface. Anthropic's documentation singles out one thing it can't change: the permission prompt.

That single exception tells you where Anthropic thinks the line is. Everything else in the session is now programmable.

Mods shipped in Claude Code v2.1.287 on October 1, 2026, and they’re on by default. The Desktop app’s bundled copy supports them from v2.1.286. In my harness checklist piece, I split an agent harness into two roles. The proposer is the model deciding what to do. The enforcer is the layer deciding whether it’s allowed. Mods are a third thing: in-process code that sits between those two, able to watch, edit, or overrule both. If you run Claude Code on a team, or just install things from GitHub, it’s worth understanding exactly where that code sits.

Settings hooks lived outside. Mods live inside.

Claude Code already had hooks: a shell command, HTTP request, or prompt you configure in settings.json to run on lifecycle events like PreToolUse or Stop. They're useful, but each one is a separate program. Claude Code runs it, reads its answer, and moves on.

A mod is a plugin whose code is a JavaScript or TypeScript module that registers event handlers. Those handlers run inside the Claude Code process. A minimal mod is three files: a plugin.json manifest, a hooks.json that points at the code, and a register.js that exports a register(on) function.

Each handler gets three arguments: $ (the mods API), e (the event, deeply frozen), and next (the rest of the chain). What you do with next decides everything:

• Observe: do some work, then return next(e). The event continues unchanged.
• Rewrite: call next with a modified copy, e.g. next({ ...e, text: e.text.trim() }) on a prompt.
• Answer: return a result without ever calling next. Nothing downstream runs, including Claude Code's own behavior.

If you’ve written Express or Koa middleware, this is that pattern applied to an agent. Here’s a small guard in that style, which refuses a curl-pipe-to-shell before it reaches the permission check:

export function register(on) {
  on('tool.call', { tool: 'Bash' }, async ($, e, next) => {
    if (/curl[^|]*\|\s*(ba|z)?sh\b/.test(e.command)) {
      // No call to next(): the command never runs, and Claude reads this text as the tool result
      return { deny: 'Piping a download straight into a shell is blocked here. Download it, show me the file, then run it.' }
    }
    return next(e)
  })
}

The { tool: 'Bash' } argument is a matcher; it can also be an array or a regex such as /^mcp__github__/. The events go well past tool calls. prompt.submit sees each prompt before the turn starts. turn.step fires before every model request and can route that request to a different model. ui.render lets a mod restyle the spinner, a tool call's row, or the question dialog.

The chain has an order, and the order is the security model

Several mods can handle the same event, so Claude Code arranges them into one middleware chain. The first handler is outermost: it sees the event before everyone else, sees the result after everyone else, and decides whether the rest run at all. For a tool call, the documented order is:

1. PreToolUse hooks from managed settings. A block here is final; no mod ever sees the call.
2. The built-in guard sec-default@builtin (where it loads), organization mods listed in prependPlugins, and any other mod that counts as the organization's.
3. Mods the user installed.
4. Organization mods listed in appendPlugins.
5. Claude Code’s other built-in mods.
6. PreToolUse hooks from every other settings file and from plugins' hooks.json, then the permission check, then the tool.

Two consequences matter.

First, step 6 is downstream of the user’s mods. A mod that answers a tool.call without calling next means your ordinary PreToolUse hooks never run for that call. And because a separate tool.check event fires after the permission rules have decided, a mod can turn an ask into an allow. In auto mode, a call a mod approves runs without a classifier check.

Second, the only things that reliably sit above user mods are managed: managed hooks, prependPlugins, and the built-in guard. Everything a developer configures personally sits below whatever mods they install.

What the built-in guard does, and doesn’t, cover

sec-default loads ahead of every user-installed mod, and users can't turn it off. But it only loads when the machine has managed settings or the user is signed in on a Team or Enterprise plan. Authenticate with an API key, or through Amazon Bedrock, Google Cloud's Agent Platform, or Microsoft Foundry, on a machine without managed settings, and there's no guard.

Where it does load, it protects what the organization manages: managed hooks, managed CLAUDE.md, the system prompt, managed MCP servers' tools and descriptions, and how other mods read settings. Deny rules also win. A user's mod can't approve a call that a deny rule refuses unless an admin sets allowModsToOverrideDenyRules.

The gap is in a different place than most people expect. Deny rules govern Claude’s tool calls. They don’t govern the mod’s own API calls. Anthropic’s admin docs say this directly: with Read(.env) denied, a mod can still read that file with $.fs.read, or start a program that does. Mods aren't sandboxed. If you enable Claude Code's sandbox, it wraps the Bash commands Claude runs, and a process a mod starts runs outside it.

Backslash Security tested this on Claude Code 2.1.291 and published the results on October 12. Their proof-of-concept mod read a test .env file that Claude Code's permission check had refused to the agent. On a Team account with the guard on, it still read the Claude token kept in the environment. A second mod rewrote a harmless-looking command into one that sent a test secret to a local listener. The transcript showed the command the agent requested, not the one that ran. None of this contradicts the documentation. It's the documented reach, demonstrated.

“You Should Know”: the mod with no hands

The most interesting built-in mod runs in the opposite direction. “You should know” (cc-plugin-you-should-know@builtin) is off by default. Turn it on with:

/plugin enable cc-plugin-you-should-know@builtin

A side agent then periodically re-reads the conversation during longer tasks. It looks for one thing you might have missed, such as a tradeoff Claude made without saying so or a result that looks off, and shows it as a note above the prompt. The design choices are what make it safe:

• No tools. The side agent is a separate request to the same model and can’t read files, run commands, or change what Claude does.
• Isolated from the main agent. Claude never sees a note unless you choose “Chat in main session.”
• Honest about cost. Each check reuses the conversation’s prompt cache, but the mod’s own instructions (several thousand tokens) and the reply are billed in full every time.

It only appears with a direct Claude API connection, either a claude.ai account or a Console key. It’s unavailable behind an LLM gateway or a Claude apps gateway, with a key from apiKeyHelper, or in sessions that don't fetch feature flags, such as one with DISABLE_TELEMETRY set.

This is the read-only proactive lane I keep pointing to in always-on agents: an observer that can advise but not act. Anthropic shipped the deepest extension surface Claude Code has had, and its own headline built-in mod deliberately uses almost none of that reach.

Locking it down

For an individual developer:

• Inspect before you install. claude plugin validate ./some-mod lists the events a mod handles (hooks:) and the API methods its code calls (calls:) without running it. Look for $.fs.write, $.process.run and $.process.spawn, $.http.fetch, $.env.get, and $.settings.read. Claude Code refuses to load a mod whose API use this static check can't read. That's what makes the list trustworthy, but it tells you which functions a mod calls, not which files or hosts it touches. Read the code too.
• Use the off switches. --safe-mode disables installed mods for a session. "disableAllHooks": true in ~/.claude/settings.json stops them everywhere, along with your settings hooks.

For an organization, the blunt instrument is one managed setting:

{
  "pluginConfigs": {
    "cc-plugin-sec-default@builtin": {
      "options": { "allowManagedModsOnly": true }
    }
  }
}

With that set, no mod a user brings loads. That covers marketplace installs, --plugin-dir, and mods Claude writes during a session. Users' settings hooks, status lines, and built-in mods keep working. The guard reads the option only from managed settings, so a user can't undo it from their own settings file.

If you’d rather admit some mods, write a policy mod. Deploy it from an admin-owned local marketplace directory and list it in prependPlugins. Its plugin.register hook sees each other mod's calls: list before that mod loads:

const RISKY = ['process.run', 'process.spawn', 'http.fetch', 'env.get']

async function vet($, e, next) {
  const hits = e.uses.calls.filter((c) => RISKY.includes(c))
  if (e.tier === 'user' && hits.length > 0) {
    return { refuse: 'Policy: user mods may not call ' + hits.join(', ') }
  }
  return next(e)
}

export function register(on) {
  // A hook that throws or times out is skipped by default (fail open).
  // The .catch handler makes this check fail closed for user mods instead.
  on('plugin.register', vet).catch(async ($, e, next) =>
    e.tier === 'user' ? { refuse: 'Policy check failed; mod not loaded' } : next(e)
  )
}

That .catch matters. By default, a hook that throws or times out is skipped, so the check fails open. Anthropic's admin docs show the same pattern for exactly this reason. Two documented gaps are still worth knowing. If the worker thread that runs installed mods crashes three times, Claude Code unloads every non-built-in mod, including yours, until a reload. And --safe-mode turns off your policy mod along with everything else.

Three weeks of hardening, read from the changelog

The changelog since 2.1.287 reads like a live threat model. Version 2.1.295 fixed a case where a mod’s hook could be handed a deeply nested tool input that had been silently cut short, so a guard could approve content it never saw. Version 2.1.299 fixed hooks receiving values with __proto__ keys silently removed, which could skip a tool.check guard. It also fixed a sandboxed shell command being able to write into a --plugin-dir folder whose code Claude Code then reloads and runs outside the sandbox, and it improved enforcement of managed rules and hooks against user-installed mods. Version 2.1.304 added decider: "person" to tool.check. A guard can now insist that only the permission dialog (or a host's approval callback) can approve a call, and the call is refused where nobody can be asked.

None of these are scandals. They’re what it looks like when a team hardens a middleware boundary in public. They also show that guard logic is subtle enough that “I wrote a deny hook” doesn’t mean “I’m protected.”

The takeaway

Mods turn Claude Code from a coding assistant with settings into a programmable agent runtime. That’s genuinely useful: cost meters, context charts, command guards, and custom panes all become afternoon projects. But the harness question changes from “what did I configure?” to “whose code runs first?”

The good news is that Anthropic built the right controls: static capability listing, a managed chain order, a fail-closed guard, and a showcase mod with no tools. The catch is that most of them only apply where someone has deployed managed settings. On a personal machine, the trust boundary is whatever you last installed.

Sources: — Anthropic, Claude Code Docs — “Mods overview” (code.claude.com/docs/en/plugins/mods/overview), accessed Oct 21, 2026 — Anthropic, Claude Code Docs — “Manage mods for your organization” (code.claude.com/docs/en/plugins/mods/admin), accessed Oct 21, 2026 — Anthropic, Claude Code Docs — “React to events with a mod” (code.claude.com/docs/en/plugins/mods/events), accessed Oct 21, 2026 — Anthropic, Claude Code Docs — “Interactive mode: You should know suggestions” (code.claude.com/docs/en/interactive-mode), accessed Oct 21, 2026 — Anthropic, Claude Code changelog, v2.1.293–v2.1.307 (code.claude.com/docs/en/changelog), accessed Oct 21, 2026 — Yossi Pik and Eli Rapaport, “Claude Code mods: What they are, and what you should be aware of,” Backslash Security blog, Oct 12, 2026