1. What is a logit ?

A logit is the raw, unscaled score output by a model say z.
The sigmoid function converts z to a model probability, p:

p = 1 / (1 + exp(−z))

The inverse calculation is:

z = log(p / (1 − p))

A zero logit gives a probability of 0.5. A positive logit gives a probability above 0.5. A negative logit gives a probability below 0.5.

For mutually exclusive classes, softmax converts the class logits to a probability distribution:

pᵢ = exp(zᵢ) / Σⱼ exp(zⱼ)

The probabilities sum to one. The difference between two logits has a useful interpretation:

zᵢ − zⱼ = log(pᵢ / pⱼ)

One multi-class logit does not give the log-odds of that class against all other classes. In exact arithmetic, sigmoid is reversible. Softmax preserves all differences between logits. It removes a common additive offset:

softmax(z + c) = softmax(z)

Thus, probability conversion does not always remove useful information. However, rounding, clipping, and retaining only the top k outputs can remove information.

2. During training use logits for loss calculation

A training objective that uses probabilities does not require the model to return probabilities. CrossEntropyLoss accepts class logits. It combines log-softmax with negative log-likelihood. BCEWithLogitsLoss combines sigmoid with binary cross-entropy. The combined calculation improves numerical stability. [1, 2]

Consider a binary prediction with z = 100 and label y = 0. The correct loss is approximately 100. An explicit sigmoid calculation can round the probability to one. The expression −log(1 − p) then becomes unstable. The equivalent logits expression avoids this problem:

L(z, y) = max(z, 0) − zy + log(1 + exp(−|z|))

Pass logits directly to CrossEntropyLoss and BCEWithLogitsLoss. An extra softmax or sigmoid changes the objective. These functions interpret their input values as logits.

3. Use scores when only order matters

Sigmoid is strictly increasing. Thus, binary logits and their probabilities give the same order in exact arithmetic. Softmax also preserves order within one vector:

argmax(z) = argmax(softmax(z))

If a process only selects the top k items, it can use the scores directly. However, softmax probabilities depend on the candidate set. Suppose candidates A and B have scores 2 and 1. Softmax assigns A a probability of 73%. Now add candidate C with score 3. A’s probability falls to 24.5%. A and B keep their scores and relative order.

These probabilities describe a distribution within the candidate set. They do not automatically give click probabilities.

4. Use probabilities for expected value

Ranking ads by click probability requires only the correct order. A higher binary logit gives a higher probability after sigmoid. Thus, logits are sufficient if the goal is to select the ad most likely to receive a click. Ranking ads by expected revenue requires the probability value. The value of a click can differ between ads:

Expected revenue per impression = P(click) × value per click.

Suppose Ad A has a 10% click probability and earns $2 per click. Its expected revenue is $0.20 per impression. Ad B has a 20% click probability and earns $0.75 per click. Its expected revenue is $0.15 per impression.

Select A to maximize expected revenue. B is more likely to receive a click, but A earns more on average. Multiplying a logit by the value per click does not calculate this average. A logit expresses log-odds, not an event probability. Probability values also let us estimate event totals. If 1,000 users each have a calibrated 10% conversion probability, the expected total is 100 conversions. More generally, add the individual conversion probabilities to estimate the expected total.

A probability threshold can also be applied directly to binary logits. For example, p ≥ 0.20 is equivalent to z ≥ log(0.20 / 0.80), or approximately −1.39. This conversion preserves the decision. However, interpreting the threshold as a 20% event chance still requires calibrated probabilities.

5. Sigmoid and Softmax != Probability

Calibration is agreement between predicted probabilities and observed event rates. Suppose a binary model assigns 0.8 to a large group of examples. For calibration, the observed event rate should be near 80% in that group. Sigmoid and softmax do not establish this agreement. They produce the required mathematical form. They do not establish that the values match actual event rates.

Temperature scaling uses:

p(T) = softmax(z / T)

Fit the positive temperature T on separate labeled data. A positive temperature changes probability values and preserves class order.

Guo et al. found temperature scaling effective on several classification benchmarks. Check calibration again when the data distribution changes. [3]. Negative sampling and class weighting can change the probability that a model learns. Before you interpret click predictions as production rates, check them against representative traffic. Apply sampling correction or calibration when required.

6. Keep logits for distillation and decoding

Distillation trains a student model from a teacher model. The teacher provides probabilities for alternative classes. Temperature controls the distribution used for training. Teacher logits let you calculate this distribution directly. Hinton et al. describe this use of soft targets. [4] Complete, exact probabilities also support temperature changes. For positive probabilities at T = 1, raise each pᵢ to 1/T and normalize the results:

qᵢ = pᵢ^(1/T) / Σⱼ pⱼ^(1/T)

However, rounded, clipped, or incomplete probabilities can lose information. Retain logits when later processes need this flexibility. In language-model decoding, greedy selection requires the highest allowed token score. Apply temperature, token masks, and score changes before normalization.

Sampling requires a probability distribution. A sampler can calculate this distribution internally. It does not have to return the full probability vector. Use log-probabilities to calculate sequence likelihoods. Add the log-probability of each token in the sequence.

7. Specify the output contract

An output contract defines what each output means. For a classifier core, I recommend logits as the default output. Provide a named probability output when required. This design supports several uses. It does not establish a speed improvement.

Figure 4. Use the output that each process requires.

Specify the output type, shape, class order, and normalization axis. Distinguish binary logits, multiclass logits, ranking scores, probabilities, and log-probabilities. Keep calibration and sampling correction versions with the model. Return both logits and probabilities only when both are required. A serving graph can include the complete calibrated probability calculation.
In summary,

1. Ranking, Regression and embedding models also return useful outputs that are not probabilities.
2. Thus, a universal probability-output rule does not apply to all models. Use logits when a process needs scores or score changes.
3. Use calibrated probabilities when a decision needs event likelihoods. Specify the conversion in the output contract.

8. References

1. PyTorch — CrossEntropyLoss
2. PyTorch — BCEWithLogitsLoss
3. Guo et al. — On Calibration of Modern Neural Networks, ICML 2017
4. Hinton et al. — Distilling the Knowledge in a Neural Network, 2015

Should models output logits or probabilities ? was originally published in Better ML on Medium, where people are continuing the conversation by highlighting and responding to this story.

Most AI agents improve because a human changes them. You add a tool, rewrite a prompt, change the control flow, or introduce a reviewer. Recent systems can automate parts of this process by searching over prompts and workflows. But the optimizer usually remains fixed and external to the agent being improved.

The Darwin Gödel Machine (DGM) asks a more ambitious question:

What if the agent that solves coding tasks also modifies the code that defines the agent?

This is often termed as self-referential agent improvement. A coding agent inspects its failures, edits its own repository, evaluates the modified version, and allows successful descendants to perform future self-modifications. But the paper’s most important contribution is not self-editing alone. It is the search strategy around self-editing.

DGM maintains an archive of many agent versions. It can revisit an imperfect ancestor, branch in a different direction, and preserve changes that are not immediately useful. This lets the system cross performance valleys that defeat greedy hill climbing. That is the central lesson: a self-improving system needs both inheritance and diversity.

1. From a Gödel machine to a Darwinian one

The original Gödel machine is a theoretical system that rewrites itself only after proving that the rewrite will improve its objective. That guarantee is largely impractical for real use-cases like coding. For a real coding agent, proving that a new tool or workflow will help across an unknown distribution of repositories is generally impossible.

A candidate self-modification is executed and evaluated on coding benchmarks. If it remains functional, it enters a growing archive of possible parents. Natural selection replaces formal verification:

1. Select an existing agent.
2. Diagnose its failures.
3. Ask it to modify its own code.
4. Test the resulting child.
5. Preserve viable children in the archive.
6. Repeat from different points in the archive.

This is a compromise : The system does not know that a change is universally beneficial. It only has empirical evidence that the modified agent performs better on a chosen evaluation.

2. What actually evolves?

The underlying foundation models remain frozen. What evolves is the coding-agent scaffold around them such as:

• File-viewing and editing tools
• Prompts and instructions
• Retry and validation logic
• Context-window management
• Candidate generation and ranking.

The initial agent is simple. It has a Bash tool and a basic editor that can view or replace entire files. Over successive generations, DGM discovers practical improvements that experienced agent engineers would recognize.

It adds line-range file viewing and precise string replacement. It validates that patches are non-empty and retries failed edits. It summarizes history when the context window is exhausted. It generates multiple candidate patches and uses another model to rank them.

Later versions condition new attempts on earlier failures. These are not exotic new reasoning algorithms. They are good pieces of agent engineering. The notable result is that the system discovers and combines them automatically.

3. Why this is self-referential ?

Suppose one generation acquires a better editing tool. That tool helps it solve benchmark tasks, but it also helps the agent edit its own repository more accurately in the next generation. A better context-management strategy similarly helps with both downstream coding and future self-modification. The improvement can therefore compound:

Better agent code produces a better self-editor, which can produce better agent code.

This closes a self-referential loop at the agent-scaffold level. The recursion is still incomplete. A separate fixed model diagnoses improvement opportunities from evaluation logs. The archive policy and parent-selection algorithm are also fixed. DGM evolves the coding agent, not the full evolutionary process that surrounds it. It is a bounded self-modifying subsystem inside a human-designed experimental loop.

4. Maintaining agent archive is the real change

A naive self-improving agent would keep only its latest or best version. The policy would look like:

Modify the current agent. Keep the change if the score improves. Otherwise revert.

4.1 The flaw in monotonic optimization

Monotonic hill climbing discards any change that doesn’t yield an immediate gain. However, major breakthroughs: like adding new tools, reviewer steps, or context summarization, often cause temporary performance drops before their full benefits are realized. Single, isolated adjustments frequently look harmful until combined with follow-up changes.

4.2 How DGM avoids it ?

The Darwinian Gödel Machine (DGM) avoids this trap by keeping promising lineages alive:

• Balanced Selection: It selects parent agents using high benchmark scores (exploitation) while giving a bonus to lineages with fewer explored children (exploration).
• Non-Monotonic Progress: This allowed DGM’s top-performing SWE-bench agent to survive two consecutive performance drops: a path a greedy optimizer would have prematurely abandoned.

Like natural selection, DGM does not require every mutation to be an instant upgrade. It relies on variation, inheritance, selection, and preserved diversity to allow complex, multi-step innovations to emerge over time.

To understand why DGM works, the authors compared Full DGM against a Fixed Modifying Agent baseline:

These results support two separate claims.

1. Allowing improved agents to perform later self-modifications works better than using the same fixed meta-agent throughout the search.
2. Preserving multiple lineages works better than greedily extending only the latest or best agent.

The discovered scaffolds also transferred to other foundation models and task slices, although the gains varied substantially. This is useful evidence that DGM did not merely overfit one exact model-agent pairing.

5. Lesson for autoresearch systems: Overcoming the valley trap

Most automated research and AutoML loops rely on monotonic hill climbing: they propose a change, test it, and discard it unless performance instantly improves. This greedy approach fails when breakthroughs require coordinated, multi-step changes where individual steps look like downgrades on their own:

Example (Improving a transformer model):

• Increasing sequence length alone => Harder optimization.
• Expanding the model backbone alone => Overfitting.
• Lowering the learning rate alone => Too slow to converge.
• Combining all three changes together => Better performance.

A standard optimizer rejects every step into this configuration. DGM’s archive-based search preserves these intermediate “failures” until the missing pieces come together.

6. How DGM samples from its archive ?

Prefer capable agents, but reduce the priority of agents that have already produced many children.

DGM normalizes weights into a probability distribution and samples parents with replacement. It uses two parent samples per iteration on SWE-bench and four on Polyglot. Each selected parent modifies a copy of its own implementation. A functioning child can enter the archive even when it performs worse than its parent.

Example recipe

Agent D has better accuracy. But it has already produced five functioning children. Agent B is approximately 3.2 times (0.269/0.083) more likely to be sampled before normalization against the rest of the archive. A possible trajectory is:

Agent B: 40%
Precise substring editing
        ↓
Agent B1: 37%
Add patch validation and retry logic
        ↓
Agent B2: 47%
Generate several patches and rank them

B1 is initially worse than B, but remains viable and enters the archive. Because B1 is underexplored, DGM may sample it later. A complementary modification can then turn its initially harmful mechanism into a stronger agent. DGM’s archive policy simply ensures that the 37% stepping stone is not permanently eliminated.

DGM provides a recipe for self-improving AI: a process driven by diverse hypotheses, parallel branches, preserved failures, and intelligent selection.

What do Self-Improving Agents learn from their mistakes ? was originally published in Better ML on Medium, where people are continuing the conversation by highlighting and responding to this story.

Imagine an agent trying to fix a software bug. After ten minutes, it has a plausible diagnosis, a patch, and one stubbornly failing test. You can give it another ten minutes. What should it do with them?

It could keep working from the same hypothesis. It could start a few fresh attempts and hope one takes a better path. It could run more tests, inspect logs, or search the documentation. Or it could stop generating ideas and spend the time checking the candidates it already has.

All four choices count as more inference compute. They do not buy the same thing.

That distinction tends to disappear when models are compared by a single accuracy or pass-rate number. The score tells us how often a system succeeded, but not how much work it was allowed to do before answering or how that work was organized. A million tokens poured into one long trajectory is a very different system from a hundred short attempts run in parallel. A system that generates five patches and tests them carefully is different again.

Given a fixed budget, where should an agent spend its next unit of compute?

One budget, four different bets

For most agent workloads, extra compute goes to some mixture of depth, breadth, interaction, and checking. It helps to look at each one separately.

Think longer

The simplest option is to extend the current trajectory: inspect another file, refine the hypothesis, make an edit, run a test, and revise. This is a good fit when progress is cumulative. In debugging, for example, the result of one test often determines the next useful test. Restarting from scratch would throw away context the agent worked to acquire.

The risk is path dependence. If the agent misunderstood the problem early, more tokens can make it more committed to the wrong explanation. Long reasoning traces often look productive because they contain many steps. That does not mean those steps are moving in the right direction.

Depth works best when the environment provides frequent feedback and the agent is actually using it. Without that feedback, “think longer” can become an expensive way to elaborate the first guess.

Branch wider

Instead of extending one attempt, the system can launch several independent ones. One branch might investigate a race condition; another might suspect a schema mismatch; a third might trace a recent dependency change. This is the intuition behind best-of-N sampling: diversity increases the chance that at least one attempt finds a good path.

Branching is attractive when there are several plausible hypotheses, individual attempts are cheap, and parallel hardware is available. Ten one-minute attempts may have roughly the same total trajectory time as one ten-minute attempt, yet return in close to one minute.

But breadth introduces a selection problem. Producing the right answer somewhere in a batch is not enough. The system still has to recognize it. A weak selector can erase most of the gain from sampling, which is why best-of-N results should always be read alongside the quality of the verifier.

Use tools

Agents can also spend their budget by interacting with the world: running code, querying a database, inspecting logs, browsing documentation, or calling a simulator. A useful tool result can be worth far more than another page of internal reasoning. A compiler error can kill a bad hypothesis immediately. A unit test can expose an edge case that a fluent explanation missed.

Tools are valuable because they add information. They also add latency, infrastructure cost, security concerns, and new ways to fail. An agent that reaches for a tool after every uncertain thought may spend most of its budget collecting evidence that never changes the answer. The relevant question is not whether a tool call is available, but whether its result is likely to change the next decision.

Verify harder

The fourth option is to spend compute on evaluation rather than generation. The system might run tests against candidate patches, check mathematical constraints, ask another model to critique an answer, apply static analysis, or compare several solutions with a judge model.

This is often where an apparently capable agent breaks down. Suppose ten generated patches include one correct fix. From the generator’s point of view, the task has been solved. If the verifier ranks a broken patch first, the deployed system still fails.

I find it useful to separate two quantities: the probability that a correct answer was generated, and the probability that the system selected it. The distance between them is the generation–verification gap. As candidate generation becomes cheaper, that gap becomes harder to ignore.

The best checks usually introduce evidence the generator did not already have: hidden tests, formal constraints, retrieved sources, deterministic rules, or a genuinely independent model. Asking the same model to reread its own answer can help, but agreement is weak evidence when both passes share the same blind spot.

Compute is not a single number

“Use more compute” sounds precise until a system has to be deployed. In practice, builders work with several budgets at once:

• Tokens: how much context and output the system may consume.
• Money: the acceptable cost per task or per successful result.
• Latency: how long the user can wait.
• Tool calls: how many searches, executions, or external requests are allowed.
• Concurrency: how many trajectories can run at the same time.

These constraints are not interchangeable. One ten-minute trajectory and ten one-minute trajectories may consume similar total compute, but they feel completely different to a user if the shorter attempts run in parallel. The parallel version spends more concurrency to buy lower wall-clock latency. The sequential version preserves context and may make better use of intermediate feedback.

This is why there is rarely one best configuration. Real systems live on a frontier: improving quality may cost more money; cutting latency may require more parallel hardware; stronger verification may reduce errors while delaying the response. The right point depends on the product.

Match the strategy to the task

Long sequential work is most useful when later steps depend heavily on earlier observations. Repository exploration, multi-step data analysis, interactive theorem proving, and debugging with repeated test execution all have this shape. Each useful observation narrows the next move, so a fresh branch would lose valuable state.

Breadth is a better bet when the solution space contains genuinely different hypotheses and attempts are only weakly dependent on one another. Candidate code patches, competition-style math solutions, alternate query interpretations, and planning problems often benefit from this treatment. The caveat is the same every time: diversity without reliable selection is just noise.

Verification deserves a larger share of the budget when errors are expensive and correctness can be tested objectively. Code is unusually friendly here because patches can be compiled and run. Some mathematical answers admit symbolic or numerical checks. Database queries can be checked against schemas and constraints.

Open-ended research summaries are harder. A response can be polished, internally consistent, and subtly wrong. Two model judges may agree because they learned the same shortcut. In those domains, verification should lean on external sources and explicit evidence instead of stylistic confidence.

The hardest tasks may be a poor investment

It is tempting to route the most compute to the hardest problems. That is not always where the return is highest.

Easy tasks usually need one short attempt. Extra branches mostly reproduce the same correct answer. At the opposite extreme, a task may require knowledge, tools, or representations the model simply does not have. One hundred variations of the same broken approach will not cross that gap.

The best return often comes from the middle: tasks near the system’s capability boundary. A little more investigation might uncover the missing fact. A second branch might escape a local pattern. One decisive test might distinguish two plausible patches.

That makes compute allocation an estimation problem. Is the current answer probably correct? Is the trajectory still making measurable progress? Would a new branch explore something different? Can a check resolve the remaining uncertainty? At what point is the expected improvement smaller than the expected cost?

Fixed recipes are easy. Adaptive routing is better.

Rules such as “always generate eight candidates” or “always run three critique rounds” are convenient, especially in a benchmark. They are also wasteful. Most tasks do not need the same treatment.

A practical controller can start cheaply and escalate only when the evidence calls for it:

1. Run one inexpensive attempt.
2. Apply a lightweight check.
3. Stop if the result is clearly good enough.
4. If the failure is localized, continue the current trajectory and revise.
5. If the agent appears stuck or candidates disagree, branch or invoke a stronger verifier.

That controller is doing more than solving the task. It is choosing an economical way to solve it. In a mixed workload, this scheduling policy may matter as much as the underlying model.

Benchmarks should show curves, not just scores

A single success rate hides all of these choices. A system that solves 50 percent of tasks for $0.20 each is not equivalent to one that solves 52 percent for $20. A response in thirty seconds is not equivalent to the same response after forty minutes.

Useful agent evaluations should report performance across several budget levels. At minimum, I would want to see success versus cost, tokens, wall-clock latency, and tool calls.

Tail latency and cost per solved task matter too. “Agent X scored 48 percent” tells a deployment team very little. “Agent X reached 40 percent at $0.30, 46 percent at $1.20, and 48 percent at $5, with most gains arriving before the second budget increase” is a much more useful result.

An experiment worth running

A clean comparison would hold the base model and task set fixed, then vary only the inference strategy. On a software-engineering benchmark, for example, compare five systems:

• Single trajectory: one long attempt.
• Sequential refinement: edit, test, and revise repeatedly.
• Parallel sampling: several shorter independent attempts.
• Verification-heavy: fewer candidates with more testing and ranking.
• Adaptive: branch or verify only when the current evidence calls for it.

Run the comparison once under an equal monetary or token budget, again under an equal wall-clock deadline, and again with the same number of tool calls. The winner will probably change. Sequential refinement should do well when tests provide useful intermediate feedback. Sampling should help when different hypotheses matter. Verification-heavy systems should shine when correctness is objectively testable. An adaptive policy should be the most robust across a mixed task set.

The interesting output is not a champion. It is the shape of the frontier and the conditions under which each strategy earns its cost.

The system matters more than the model name

An agent’s capability comes from the whole inference system: the model, prompt, tools, search strategy, verifier, stopping rule, scheduler, and budget. Changing any one of them can change the result.

A smaller model with a disciplined execution loop and reliable tests can outperform a stronger model used in a single pass. The reverse is also true: an elaborate agent loop can burn through tokens and tool calls while adding almost no value.

More inference compute helps when it is spent on the bottleneck. Sometimes that means staying with a promising line of reasoning. Sometimes it means starting over in parallel. Sometimes the right move is to stop generating and check the work. And sometimes the answer is simply to stop.

The next generation of agents will not be defined only by how long they can think. The more important skill will be knowing when to continue, when to explore, when to verify, and when another unit of compute is no longer worth buying.

How frontier systems allocate test-time compute

Public disclosures do not reveal the complete production schedulers behind ChatGPT, DeepSeek, Gemini, or Claude. They do reveal enough pieces to see a common pattern: route first, reason with a task-dependent budget, use tools when they can add information, and spend verification compute when several plausible answers remain.

• Compute-optimal test-time scaling. Snell and colleagues estimate prompt difficulty and change the allocation between sequential revision, parallel sampling, and verifier-guided search. Easy questions often favor revision; harder but still tractable questions benefit from a mixture of sequential and parallel work. Their prompt-dependent policy used up to four times less compute than a uniform best-of-N baseline for comparable performance.
• DeepSeek-R1. DeepSeek trains the model with reinforcement learning so that longer reasoning, reflection, and self-correction emerge inside a single trajectory. It can also spend more compute outside that trajectory by sampling several solutions and applying majority voting. The paper reports that majority voting raised DeepSeek-R1-Zero’s AIME 2024 score from 71.0 to 86.7 percent. It does not describe a deployed controller that dynamically chooses the reasoning length or number of samples for every request.
• DeepSeek-GRM. This work allocates extra compute to verification. A generative reward model produces task-specific principles and critiques; parallel sampling creates several evaluations; and a meta-reward model guides the final vote. In other words, the additional budget is spent making the judge stronger rather than merely generating more candidate answers.
• GPT-5 in ChatGPT. OpenAI describes a real-time router that chooses between a fast model and a deeper reasoning model. The decision uses conversation type, apparent complexity, tool requirements, and explicit intent such as a request to “think hard.” This is the clearest public example of a product-level compute controller: simple requests take the cheaper path, while harder requests are routed to a model allowed to spend more inference compute.
• OpenAI o3 and o4-mini. These models allocate compute within a longer reasoning trajectory and across tool calls. Reinforcement learning teaches them not only how to call web search, Python, file analysis, and other tools, but when those calls are useful. OpenAI also exposes reasoning-effort settings, which let the surrounding system trade latency and cost for deeper reasoning. The exact internal stopping policy remains private.
• OpenAI deep research. Deep research spends its budget on a long, environment-grounded trajectory. It searches, reads sources, uses Python, backtracks, and changes direction in response to evidence. OpenAI reports performance as a function of maximum tool calls, making the tool budget an explicit axis of test-time scaling. The public description explains the learned behavior but not the precise rule that decides when the research is complete.
• Gemini dynamic thinking. Google documents a dynamic mode that adjusts reasoning effort according to request complexity. Developers can also choose explicit thinking levels when they need tighter control over latency, cost, or reasoning depth. This makes the allocation decision partly model-driven and partly a product or application policy.
• Claude adaptive thinking. Anthropic’s adaptive-thinking mode lets Claude decide when extended reasoning is useful, while effort levels make the system more or less selective about using it. For work that separates cleanly, Claude’s agent teams add a second allocation mechanism: multiple agents can run independent subtasks in parallel.

The research ideas underneath these systems

Several earlier papers isolate the mechanisms now appearing together in frontier products. Training Verifiers to Solve Math Word Problems generates many solutions and spends additional compute selecting one with a learned verifier. Self-Consistency samples diverse reasoning paths and chooses the answer with the strongest agreement. Tree of Thoughts branches, scores, and backtracks over intermediate reasoning states. ReAct alternates reasoning with actions so that observations from tools can change the plan.

Taken together, these systems point toward a layered design: an outer router selects the mode or model, the reasoning model controls depth, an agent loop decides whether to seek external evidence, and a verifier or voting procedure resolves disagreement. The broad design is visible. The thresholds, confidence models, stopping rules, and production economics are still mostly proprietary.

Agent Compute: Think Longer, Branch Wider, or Verify Harder ? was originally published in Better ML on Medium, where people are continuing the conversation by highlighting and responding to this story.

We are moving away from models designed for isolated pattern recognition and toward unified architectures capable of sequential decision-making. At the heart of this transition is a powerful, highly generalized recipe: Reinforcement Learning (RL) combined with Transformer architectures.

Together, these technologies turn disparate, complex tasks into a singular, repeatable pattern. By observing a state, reasoning over context, executing an action, and updating policies based on environmental feedback, we can build robust, general-purpose problem solvers.

1. The Transformer: A Programmable Substrate

To build a general solver, you first need a flexible model capable of unifying diverse data modalities. The Transformer serves as this programmable substrate.

By mapping complex inputs and outputs to a shared token vocabulary, Transformers establish text — or, more broadly, tokens — as a universal interface. This interface can elegantly represent:

• States: The current context of an environment or problem.
• Actions: Code to execute, API calls to make, or robotic movements to trigger.
• Plans & Programs: Long-term strategic reasoning or workflow definitions.
• Feedback: Natural language critiques, error logs, or compiler warnings.

Because Transformers are adept at in-context learning and sequence modeling, they are perfectly suited to read these representations, maintain an understanding of the state, and generate subsequent tokens that dictate the next logical action.

2. The Limits of Next-Token Prediction

Despite their flexibility, Transformers trained solely on next-token prediction are fundamentally limited when it comes to long-horizon problem-solving. Next-token prediction is an exercise in imitation; it aims to mimic the statistical distribution of the training data.

While it can approximate reasoning, it does not inherently provide goal-directed pressure. When an agent must navigate a complex environment, write and debug functional code, or prove a mathematical theorem, success is strictly defined by the final outcome — not by how mathematically probable the sequence of text was. To achieve true problem-solving capabilities, the model must transition from passive imitation to active optimization.

3. Reinforcement Learning: The Goal-Directed Engine

Reinforcement Learning supplies the optimization loop that next-token prediction lacks. Instead of training a model on static, isolated examples, RL places the Transformer in dynamic environments where it must:

1. Try an action (or sequence of actions).
2. Interact with an environment (e.g., a compiler, a game engine, or a web browser).
3. Receive scalar rewards or verifier feedback.
4. Improve its behavior over full trajectories.

In this architecture, the Transformer is the engine, while RL acts as the steering wheel, aggressively optimizing for outcomes. This combination is particularly potent for autonomous agents, ML engineering tasks, and tool use, where the definition of “correct” is grounded in execution rather than linguistic fluency.

4. Overcoming Non-Differentiability with TextGrad

A significant hurdle in applying RL to text-based environments and tool use is the non-differentiable nature of the environment. You cannot easily backpropagate an error signal through an external compiler or a black-box API.

This introduces the need to frame text-representable tasks as optimization problems using TextGrad-based reinforcement learning. TextGrad solves the credit assignment problem over non-differentiable trajectories by using language itself as the gradient.

In a TextGrad-powered RL loop, the process looks like this:

• Generation: An agent proposes a textual artifact (e.g., a block of code, a plan, a model configuration).
• Execution: The artifact is executed in a tool-mediated environment.
• Evaluation: The system receives scalar rewards from automated tests or verifiers.
• Textual Credit Assignment: Instead of numerical gradients, the system generates textual gradients. A verifier model explains the specific source of failure in natural language and suggests local, targeted repairs.
• Optimization: The RL loop continues to drive exploration and reward maximization, while TextGrad provides the high-fidelity, natural-language feedback necessary to iteratively optimize prompts, policies, research workflows, and action sequences.

Example: Application to a database query task, broken down into exactly five steps.

1. Generation: An AI agent is tasked with finding the top five highest-spending customers and generates an initial textual artifact (a SQL query): SELECT name FROM customers ORDER BY revenue LIMIT 5;
2. Execution: The query is run against an external, black-box database engine — a non-differentiable environment where standard backpropagation cannot be used.
3. Evaluation: The database execution fails and returns an error message (ERROR: column "revenue" does not exist), resulting in the system assigning the attempt a scalar reward of 0 (failure).
4. Textual Credit Assignment: A verifier model generates a “textual gradient” to explain the failure and suggest a targeted repair: “Your query failed because the database column is named ‘total_sales’, not ‘revenue’, and you must add ‘DESC’ to sort by the highest spenders.”
5. Optimization: The agent uses this natural language feedback to update its approach, generating a new, successful query (SELECT name FROM customers ORDER BY total_sales DESC LIMIT 5;) and earning a reward of 1.

5. A different Paradigm for Optimization

Next token prediction is incredibly powerful, but this approach is inherently limited when we ask AI to step outside of text generation and interact with external systems. As we shift toward autonomous, agentic workflows, a new framework has emerged to bridge this gap: the TextGrad-powered Reinforcement Learning (RL) loop.

Here is how these two paradigms fundamentally differ, and why this shift is so critical.

1. From Guessing to Achieving (The Objective)

Instead of optimizing for statistical likelihood, Textgrad + RL is explicitly designed to maximize outcome-based rewards, ensuring the agent’s outputs actually achieve a tangible goal.

2. Text as an Action, Not Just an Output (The Interface)

In standard models, the interface is simply natural language strings; you provide text, and the model returns text. In a TextGrad RL loop, text transforms into executable actions and states. A generated string isn’t just an essay or a poem; it’s a piece of code to be compiled, a database query to run, or an actionable step in a larger plan.

3. Language as the Gradient (The Feedback Mechanism)

Standard models learn through mathematical backpropagation, adjusting numerical weights against ground-truth training data. However, you cannot backpropagate through a black-box API or a live software environment. TextGrad solves this by replacing mathematical gradients with verifier rewards and textual gradients. When an action fails, the system doesn’t rely on a simple numerical penalty; instead, a verifier model uses natural language to explain exactly why it failed and how to fix it on the next attempt.

4. Navigating Complexity (The Problem Horizon)

Next-token prediction excels in short horizons, perfectly suited for handling isolated, single-turn prompts and responses. Conversely, TextGrad-RL is built for the long game. It enables AI agents to navigate complex, multi-step trajectories, retaining context, processing feedback, and adjusting strategies over numerous iterations until the overarching task is complete.

Ultimately, traditional next-token models remain the undisputed champions of knowledge retrieval and drafting. But when you need a system capable of reliable execution, iterative debugging, and autonomous solving, the TextGrad + RL loop provides the necessary architecture, giving AI the feedback mechanisms it needs to interact successfully with the real world.

The Takeaway: By marrying the flexible substrate of Transformers with the goal-directed optimization of RL and the interpretability of TextGrad, we are laying the foundational architecture for the next generation of autonomous problem solvers.

Language as the Gradient: From text generation to autonomous execution was originally published in Better ML on Medium, where people are continuing the conversation by highlighting and responding to this story.

In traditional search architecture, using multiple retrievers often meant getting slightly different flavors of the same results. A keyword retriever might find a document based on its title, while a graph retriever might find the exact same document because it was heavily cited. There was usually a significant overlap in the retrieved lists.

However, as Small / Large Language Models (SLMs / LLMs) and advanced neural retrieval systems have entered the stack, this overlap is shrinking rapidly. The documents retrieved by different sources are becoming fundamentally different, and this divergence is increasing as LLMs unlock deep semantic connections. Here is why different retrieval sources diverge, and why LLMs are accelerating this trend.

2. The Baseline: Mechanical Differences in Retrieval

Even before LLMs, different systems naturally pulled different documents because they looked for entirely different signals:

• Lexical Retrievers (BM25): Look for exact vocabulary matches and term frequency. They are highly literal.
• Knowledge Graphs: Look for structured relationships (e.g., “Company X is owned by Parent Y”).
• Collaborative Filtering: Looks at human behavioral signals (e.g., “Users who clicked this also clicked that”).

A lexical search for “fixing a leaking pipe” will heavily bias toward DIY articles that repeat the words “fix,” “leak,” and “pipe.”

3. The LLM Shift: From Vocabulary to Deep Intent

The introduction of LLM-based retrieval: such as advanced cross-encoders, late-interaction models with LM (language models), and GR (generative retrieval): shifted search from matching text to matching intent.

As LLMs develop a deeper understanding of semantic connections, the lists they generate look increasingly alien compared to traditional keyword lists.

This happens in three distinct ways:

1. The Zero-Overlap Phenomenon LLMs can identify perfect answers that share absolutely no keywords with the user’s query. If a user searches for “why is my coffee tasting excessively bitter and metallic,” a lexical search will look for “bitter” and “metallic.” An LLM understands the underlying chemical process of over-extraction or burr grinder friction. It might retrieve a highly technical barista manual about “adjusting extraction yield and grind uniformity.” The lexical and semantic retrieval lists will have near-zero overlap, yet both might be useful.

2. Domain specialization of embeddings: Traditional dense embeddings often struggle with words that have multiple meanings, relying on broad proximity. Modern LLMs process the entire sequence of a query to lock in the exact context. For a query like “Python performance in tight loops,” an LLM immediately discards any zoological context and retrieves highly specific software optimization guides. A weaker retriever might still pull a mix of programming and snake-related documents if the exact keyword overlaps are strong enough.

3. Implicit Problem Solving: Humans often search for symptoms, but they want cures. LLMs bridge this gap through multi-hop semantic reasoning. If an employee searches the corporate intranet for “can’t access HR portal 403 error,” a keyword engine retrieves the IT manual page for “403 Errors.” An LLM-based retriever understands that a 403 on the HR portal usually indicates an expired VPN session, and retrieves the “How to reset your VPN token” document.

4. The Consequence for Recsys Architecture

As LLMs become better at uncovering these deep, non-obvious connections, the disparity between retrieval lists is expanding.

This growing gap is exactly why rank fusion has become the backbone of modern search. A system can no longer just assume that a document is highly relevant because it appears in every retriever’s list. Instead, the architecture must trust that the LLM is finding profound semantic matches, while relying on the lexical retriever to ground the search and prevent AI hallucinations.

By combining highly literal, keyword-dense documents with deep, conceptually-matched documents, systems can deliver a final ranked list that satisfies both the mechanics of the user’s query and the true intent behind it.

So, this boils down to one critical question:

How do we merge multiple ranked lists into one better ranking?

This seemingly simple problem has driven nearly three decades of research. Solutions have evolved from simple voting schemes to fully learned ranking systems that optimize business metrics directly. You can think of this evolution as five successive levels of sophistication, moving from simplicity to high-quality, data-driven complexity.

5. Why Rank Fusion Exists ?

Imagine searching for: “best restaurants near Stanford serving ramen.”

• Lexical retriever (BM25): Excels at exact keyword matches.
• Dense retriever: Understands semantic similarity and context.
• Location retriever: Knows geographic relevance.
• Popularity model: Knows user preferences and trends.

Each system retrieves useful — but entirely different — documents. Rank fusion solves the problem of combining these disparate results to produce one superior ranking.

5.1 Baseline: Single Retriever

The simplest system doesn’t fuse anything. You simply choose one retrieval method (e.g., query to BM25, or query to Dense Retriever) and return the ranked list.

• Advantages: It features the simplest architecture, is very fast, and is easy to debug.
• Disadvantages: It misses complementary information from other methods, and a single retrieval failure ruins recall entirely.

This was the standard for classical search engines, but modern demands quickly outgrew it.

5.2 Level 1: Unsupervised Fusion

The first major advance was realizing that agreement between retrievers is highly valuable. Instead of trusting raw scores — which often aren’t directly comparable across different algorithms — we combine the rankings themselves. This family of methods requires no training data.

Reciprocal Rank Fusion (RRF)

Instead of using scores, every document receives a value based on its position in the list:

1. Where r_i(d) is the rank from retriever i, and k is a constant (usually 60). High-ranked documents consistently appearing across multiple retrievers naturally float upward. The beauty of RRF is that score normalization is completely unnecessary.
2. If a document is not retrieved by system i, its rank r_i is treated as infinity. Consequently, the fraction rrf evaluates to 0. The document simply gets no points from that retriever, but it keeps whatever score it earned from the systems that did retrieve it.
3. Some implementations assign a default penalty rank. If a retriever only returns the top K documents (e.g., Top 100), any document not in that list is assumed to be at rank K + 1 (e.g., Rank 101) or is given the rank of the total corpus size. This is particularly common when training Learning-to-Rank (LTR) models, where a feature column cannot simply be left blank (NaN) and requires a fixed numeric value.

Unsupervised methods are production-friendly, robust, and require no labels. They are surprisingly difficult to beat, which explains why RRF remains extremely popular.

5.3 Level 2: Query adaptive Fusion

Modern systems recognize that one fusion strategy doesn’t fit every query. Some queries are lexical (“RTX 5090”), while others are semantic (“movies about loneliness”). Adaptive fusion changes the approach dynamically.

• Dynamic Alpha: Predict a specific weight based on the query itself, assigning higher lexical weights for exact-match searches and higher dense weights for conceptual searches.
• Query Routers: Rather than combining everything, a router chooses which retrievers should even participate.
• Weighted RRF: Instead of equal votes, assign retriever-specific weights to the RRF calculation:

Adaptive fusion is query-aware, robust, and significantly improves long-tail performance.

5.4 Level 3: Learned Fusion

Eventually, we stop hand-designing fusion rules. Instead, the system learns exactly how rankings should be merged by directly optimizing for relevance.

• Learning to Rank (LTR): The classic production solution. Features like BM25 score, dense similarity, PageRank, popularity, and click history are fed into a model (like GBDT or LambdaMART) to predict the final ranking.
• Neural Rerankers: A cross-encoder jointly processes the query and candidate document using a Transformer to generate a highly accurate relevance score. It is much more accurate, though significantly slower.
• Fusion-in-T5 (FiD): Multiple retrieved documents are encoded independently and fused inside the transformer. This architecture is highly influential in modern Retrieval-Augmented Generation (RAG).
• Production KPI Rankers: Large companies optimize directly for business metrics such as Click-Through Rate (CTR), dwell time, purchases, or user satisfaction (Level 4).

Most modern recsys systems don’t just pick one level of rank fusion, they stack them. A typical production pipeline starts with multiple retrievers, uses Level 1 (RRF) for a fast initial merge, applies Level 2(Adaptive Routing), and refines the candidates using Level 3(Neural Rerankers) before making a final pass for business KPIs Level 4.

This layered approach balances robustness and relevance. The evolution of rank fusion reflects a broader industry shift from handcrafted heuristics toward data-driven AI systems.

The Retrieval Divergence Problem: Why Rank Fusion Matters More in the LLM Era ? was originally published in Better ML on Medium, where people are continuing the conversation by highlighting and responding to this story.

1. Self-improving agents need a mechanism for converting experience into durable improvements. A typical agent only accumulates logs: it records failures, retries with more context, and eventually suffers from context bloat. TextGrad provides a more structured alternative. It treats parts of the agent harness as optimizable variables and uses LLM-generated feedback as a gradient-like signal for improving them.
2. In a typical agent, the editable variables are not model weights. They are text based parameters like system prompts, tool-use instructions, routing rules, memory-writing policies, verifier prompts, retrieval queries, retry policies, and task decomposition templates. After an agent rollout, an evaluator inspects the result and produces textual feedback. TextGrad then routes that feedback back to the relevant variable and rewrites it.

For example, suppose a research agent fails because it retrieves too many irrelevant papers and then summarizes them without checking whether they match the user’s question. A normal retry loop might simply say, “be more careful.” TextGrad can produce a more targeted update:

Failure: The retriever returned papers that matched the query keywords but were not actually relevant to the user’s research intent.

Textual gradient for retrieval prompt:
The retrieval prompt should optimize for semantic relevance rather than surface-level keyword overlap. It should prioritize papers that directly address the target mechanism, benchmark, task, or empirical claim, and filter out papers that only use similar terminology.

Updated retrieval prompt:
Retrieve papers that directly support the user’s research question. Prefer sources that match the intended task, method, benchmark, mechanism, or empirical claim. Exclude papers that only share keywords without substantive relevance.

Reflection vs TextGrad

TextGrad is different from ordinary reflection. Reflection usually produces a generic note such as “next time, verify assumptions.” TextGrad asks for an actionable edit to a specific variable. The output is is a patch.

TextGrad is especially useful for agents because most agent behavior is controlled by text, not model weights. They can be optimized without fine-tuning the base model.

• The system prompt defines constraints. Tool descriptions define how to call tools.
• Planner prompts define decomposition behavior. Memory policies define what gets stored.
• Verifier prompts define acceptance criteria.

TextGrad Application

1. A practical self-improving agent can apply TextGrad across several parts of its workflow. It can improve the planner prompt so the agent decomposes underspecified tasks more effectively. It can refine the tool-use policy so the agent knows when to call search, code execution, calendar, file, or other external tools. It can improve the retriever prompt to select better sources and formulate more precise queries. It can strengthen the verifier prompt to catch failures more reliably and avoid false acceptance. It can update the memory policy to decide what should be stored, revised, or discarded. It can also refine the router prompt so the right sub-agent is selected for each class of task.
2. The most important use case is verifier improvement. Many agent failures are not caused by generation alone. They happen because the agent accepts a weak answer too early. TextGrad can improve the verifier by turning missed failures into stricter acceptance criteria. Example:

Failure: The verifier approved an answer that relied on outdated information.

Textual gradient for verifier: The verifier should explicitly determine whether the answer depends on facts that may change over time. When time-sensitive claims are present, it should require up-to-date sources and reject responses that rely only on the model’s memory.

Updated verifier rule: For claims involving current prices, laws, schedules, rankings, executives, software versions, or recent events, require a current source. Reject the answer when no fresh source is available.

Guardrails for TextGrad:

• TextGrad alone is not sufficient. Since the textual gradient is produced by an LLM, it may be ambiguous, incorrect, or too narrowly fitted to a single failure case.
• A reliable self-improving agent needs safeguards around the update loop. Each edit should be evaluated before and after it is applied. The agent should maintain a held-out validation set to detect regressions.
• Updates that fix one example but reduce overall performance should be rejected. Every change should be tracked with diffs and rollback points. When possible, evaluation should use an independent verifier rather than the same model that proposed the edit. The system should also favor small, targeted changes over broad prompt rewrites.

For self-improving agents, those textual control surfaces are the harness. TextGrad gives the agent a way to learn from experience without training the base model. It does not make the LLM intrinsically smarter. It makes the surrounding agent system better organized, better routed, and better verified over time.

TextGrad for Self-improving Agents was originally published in Better ML on Medium, where people are continuing the conversation by highlighting and responding to this story.

1. The LLM proposes code changes.
2. A programmatic evaluator runs the code and scores it.
3. A database keeps the best and most diverse candidates.
4. The loop repeats until the system discovers a verifiable solution — better algorithm, heuristic, kernel, or mathematical construction.

DeepMind describes it as an evolutionary coding agent that combines Gemini models, automated evaluators, and an evolutionary framework for general-purpose algorithm discovery and optimization.

Key idea is simple: let an LLM mutate code, but let execution and scoring decide what survives.

while compute_budget_remaining:
    parent, inspirations = database.sample()
    prompt = prompt_sampler.build(parent, inspirations)
    diff = llm.generate(prompt)
    child = apply_diff(parent, diff)
    results = evaluator.execute(child)
    database.add(child, results)

The important architectural point is that the LLM is not trusted by default. It proposes candidates. The evaluator decides whether they are valid and useful. This is why AlphaEvolve works best on problems where solutions can be automatically checked: algorithms, kernels, math constructions, schedulers, compilers, and other measurable systems.

1. Task specification

The user provides three things:

1. Scaffold / Research plan / An initial program.
2. Constraints: Marked regions of code that AlphaEvolve is allowed to modify.
3. An evaluation function that returns scalar scores.

In the paper, the evaluator is typically a Python evaluate function returning a dictionary of scalar metrics. The code regions to evolve are marked with comments such as # EVOLVE-BLOCK-START and # EVOLVE-BLOCK-END, so the system can modify selected parts of a larger codebase without rewriting everything blindly.

AlphaEvolve starts with a scaffold. Humans define the “what”: the problem, the constraints, the scoring function, and the editable code blocks. AlphaEvolve searches for the “how”: the actual implementation that improves the score.

# EVOLVE-BLOCK-START
def score_machine(job, machine):
    # initial heuristic
    return machine.free_cpu + machine.free_mem
# EVOLVE-BLOCK-END

def evaluate(candidate_program):
    simulation_result = run_scheduler_simulation(candidate_program)
    return {
        "jobs_placed": simulation_result.jobs_placed,
        "stranded_resources": -simulation_result.stranded_resources,
    }

AlphaEvolve would repeatedly mutate score_machine, run the scheduler simulation, and keep variants that place more jobs while wasting fewer resources.

2. Prompt sampler

• The prompt sampler builds the context given to the LLM. It does not simply say, “write better code.” It includes the current program, previous high-performing programs, scores, evaluation outputs, task instructions, and sometimes additional prompt-evolution hints.
• This matters because AlphaEvolve is doing in-context evolutionary search. The LLM sees examples of what worked before, what failed, and what objective it is optimizing. The system can also evolve meta-prompts: instructions and context suggested by the LLM itself, stored in a separate database analogous to the solution-program database.

The prompt sampler is AlphaEvolve’s memory-to-language bridge. It converts the evolutionary history into a prompt that the LLM can reason over.

3. LLM ensemble for efficient explore-exploit

AlphaEvolve uses an ensemble of Gemini models. A cheap model which contributes high-throughput exploration, while the high reasoning model contributes occasional higher-quality suggestions that can move the search forward.

This is a classic exploration-exploitation split.

AlphaEvolve treats LLM calls as mutation operators. A smaller/faster model gives lots of mutations. A stronger reasoning model gives more expensive but potentially more strategic mutations.

4. Code mutation format

For larger codebases, the LLM usually outputs changes as SEARCH/REPLACE diffs. The system applies those diffs to the current program, producing a child program. For very small programs, AlphaEvolve can instead ask the model to output the full code block.

Example:

<<<<<<< SEARCH
return machine.free_cpu + machine.free_mem
=======
cpu_residual = job.cpu / machine.free_cpu
mem_residual = job.mem / machine.free_mem
return -1.0 * (
    cpu_residual
    + mem_residual
    + mem_residual / cpu_residual
    + cpu_residual / mem_residual
)
>>>>>>> REPLACE

This style is useful because it lets AlphaEvolve operate on real codebases with constraints instead of isolated toy functions.

5. Evaluation cascades

1. The evaluator is the system’s source of truth. It executes the child program and returns scores. AlphaEvolve supports evaluation cascades, where cheap tests run first and expensive tests only run on promising candidates. It can also use LLM-generated feedback for softer properties such as simplicity, and it can parallelize expensive evaluations across clusters.
2. It also supports multiple objectives. For example, a candidate can be scored on correctness, runtime, memory use, simplicity, robustness, and generalization.
3. Multi-metric optimization can improve even a single target metric because different objectives surface structurally different programs.

The evaluator turns subjective coding into measurable search. Without it, AlphaEvolve would just be sampling plausible code. With it, every proposal becomes an experiment.

6. Long term Memory

Every candidate program is stored with its scores and outputs. The database is not just a leaderboard. Its job is to resurface useful previous ideas while balancing exploration and exploitation.

That is important because many algorithmic discoveries are not monotonic. A candidate that is not globally best may contain a useful trick that later combines with another idea.

7. Distributed async executor

AlphaEvolve is built as an asynchronous pipeline with a controller, LLM samplers, and evaluation nodes. The system optimizes for total throughput, not the latency of one candidate, because discovery depends on evaluating many ideas under a fixed compute budget.

AlphaEvolve is like a research lab running thousands of code experiments in parallel.

Application 1: faster matrix multiplication

Matrix multiplication can be represented as a tensor decomposition problem. The fewer rank-one terms in the decomposition, the fewer scalar multiplications are required.

AlphaEvolve improved the state of the art for 14 matrix multiplication targets. Most notably, it found a rank-48 algorithm for multiplying two 4 × 4 complex-valued matrices, improving over the 49 multiplications obtained by recursively applying Strassen’s 1969 algorithm in the characteristic-zero setting.

AlphaEvolve modified a search algorithm that looked for better tensor decompositions. The evaluator checked the rank and exactness. Over many iterations, the system discovered a lower-rank construction.

Application 2: Training kernel speedup

AlphaEvolve optimized tiling heuristics for an important matrix multiplication kernel used to train Gemini. The objective was actual runtime on TPU accelerators. Google reported an average 23% kernel speedup over an expert-designed heuristic and a 1% reduction in Gemini’s overall training time. The optimization time dropped from months of engineering work to days of automated experimentation.

This is recursive infrastructure optimization: Gemini powers AlphaEvolve, and AlphaEvolve finds code that makes Gemini training faster.

What makes the architecture powerful

The useful mental model is:

LLM creativity + program execution + Evolutionary memory = Scalable search

The LLM supplies semantic mutations. The evaluator supplies ground truth. The database supplies memory and diversity. The distributed pipeline supplies scale.

This avoids a common failure mode of LLM agents: plausible but unverified output. AlphaEvolve does not need the model to be right in one shot. It only needs the model to generate enough useful variation that the evaluator can select from.

AlphaEvolve: a reference architecture for turning LLMs into algorithm-discovery engines was originally published in Better ML on Medium, where people are continuing the conversation by highlighting and responding to this story.

• Differentiable top-k restores end-to-end learning by smoothing that boundary.
• Best practice: train with soft top-k, serve with hard top-k. Tradeoff is clear: optimization quality vs serving efficiency.
• This applies broadly across MoE routing, retrieval, and recommender cascades.

The problem: top-k as a broken interface

Most large-scale ML systems make a hard top-k decision: scores → pick k winners → discard the rest.

This is efficient at serving time. But during training, it is a poor interface for gradient-based optimization. The selected set changes discontinuously when scores cross, so gradients do not reflect what happens at the boundary.

Result: the model learns scores, but not the consequences of selection.

The idea: smooth the boundary

Differentiable top-k replaces the hard selection with a smooth approximation: scores → soft k-hot mask → weighted combination.

This allows gradients to flow through the selection step. At serving time, you revert to exact top-k for efficiency.

Core pattern:

• Training: scores → differentiable top-k → gradients flow
• Serving: scores → exact top-k → sparse execution

Where this shows up ?

This is the same problem across systems:

Retrieval / ranking

• top-k defines the candidate shortlist
• downstream model only sees this subset

Recommender cascades

• Multiple selection boundaries (retrieval, filtering, reranking)
• Each can break gradient flow

MoE (experts)

• Top-k decides which experts run
• Hard routing → efficient but unstable optimization

Unifying view: Selection under a budget is a learnable operator, not just a systems constraint.

Tradeoffs

1. Optimization vs efficiency

Hard top-k is a systems-optimal operator. It enforces strict sparsity, which directly translates to bounded latency and predictable resource usage. However, from an optimization standpoint it is sub-optimal.

The selection boundary is discontinuous, so gradients do not reflect how close a losing candidate was to being selected. This prevents proper credit assignment and slows or destabilizes learning.

Differentiable top-k fixes this by smoothing the boundary and allowing gradients to flow through near-miss candidates. The cost is that sparsity becomes approximate during training, which increases compute and memory footprint. In practice, this is handled by restricting the relaxation to training only and reverting to exact sparsity at serving.

2. Stability vs bias

Hard top-k preserves the exact combinatorial decision rule. There is no approximation, so the selected set is unbiased with respect to the scoring function. However, because gradients are either zero or undefined at the boundary, optimization behaves like a high-variance estimator with poor signal.

Differentiable top-k introduces a continuous relaxation, which produces stable gradients and significantly improves convergence behavior. The tradeoff is bias. The model is trained under a softened selection rule that does not exactly match the true top-k operator. This bias can lead to slight mis-calibration, especially when score gaps are small, and typically requires temperature annealing or sharpening during training.

3. Train-serve mismatch

The standard pattern trains with a soft approximation and serves with hard top-k. This creates an inherent mismatch: the model is optimized under a relaxed operator but deployed under a discrete one.

In most cases this works well because the relaxation is designed to approximate the true operator closely. However, edge cases appear when the score distribution is flat or highly competitive. Small differences that matter under hard top-k can be washed out during training. This results in minor distribution shift at inference. Mitigations include temperature annealing, straight-through estimators, or hybrid objectives that explicitly penalize discrepancies between soft and hard selections.

4. Systems complexity

Hard top-k is trivial to implement and heavily optimized across hardware and frameworks. It integrates cleanly with sparse execution engines such as MoE dispatch or retrieval systems. Differentiable top-k, in contrast, requires non-trivial operators such as Gumbel-softmax sampling or differentiable sorting relaxations.

These introduce additional numerical considerations, memory overhead, and scaling challenges, especially in distributed settings.

Efficient implementations often rely on approximations or low-rank tricks, and care must be taken to avoid turning the routing step into the dominant cost. As a result, differentiable top-k is best viewed as a training-time construct rather than a systems primitive for serving.

Industrial applications/tests

• DSelect-k: large recommender testing → significant offline gains
• Differentiable top-k ranking: measurable online revenue lift.
• Switch Transformer: confirms why hard sparsity is operationally critical.

Takeaway: There is real business value in learning through the selection boundary.

Takeaway

Routing is not just scheduling. It is model behavior. MoE gates, retrieval shortlists, and recommender filters all collapse into the same problem:

Learn the boundary, then enforce it efficiently.

• Training: differentiable approximation
• Serving: exact sparse top-k

This gives: better optimization at same serving cost.

Differentiable Top-k Routing was originally published in Better ML on Medium, where people are continuing the conversation by highlighting and responding to this story.

The paper Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity proposes a different approach. Instead of estimating model size from serving cost, it estimates size from how much long-tail factual knowledge a model appears to store.

Some facts cannot be derived through reasoning. They must be memorized. If a model reliably recalls more rare facts, it likely has higher effective parametric capacity. The paper validates this idea on 89 open-weight models ranging from 135M to 1.6T parameters and reports a strong log-linear relationship between factual recall and parameter count. These facts are evaluated using Incompressible Knowledge Probes (IKPs), a 1,400-question benchmark spanning seven tiers (T1–T7) of increasing obscurity.

The Problem: We Do Not Know How Big Closed Models Are

A common workaround is to estimate model size from inference economics: serving cost, throughput, latency, and hardware assumptions. These signals are not intrinsic. A small model served inefficiently can appear large, while a large model with aggressive optimizations can appear smaller.

IKP reframes the question. Instead of asking “How expensive is this model to run?”, it asks “How far into the factual long tail can this model recall?”.

This shifts the focus from cost to stored factual knowledge.

Compressible vs. Incompressible Knowledge

• Compressible skills: reasoning, coding style, instruction-following. These can be improved through better training procedures and synthetic data.
• Incompressible facts: long-tail associations such as obscure founding years or researcher subfields. These must be stored in the parameters.

IKP targets the second category.

Methodology: Build Facts That Models Cannot Fake by Reasoning

The benchmark contains 1,400 probes, with 200 questions per tier (T1–T7).

• T1 contains common facts.
• T7 contains extreme long-tail facts.

The dataset spans 16 domains, including geography, history, science, and researcher metadata.

Probe construction has two stages:

1. LLM-generated probes (easy tiers)
   Used primarily for T1–T2. LLMs can generate common factual questions but struggle beyond their own knowledge horizon. This stage contributes 401 probes.
2. External corpora (hard tiers)

• Wikidata: 557 probes (entity attributes, founding years)
• DBLP / arXiv / OpenAlex: 345 researcher probes

Researcher probes require both the correct subfield and a verifiable artifact such as a paper or co-author. This discourages plausible but fabricated answers.

Filtering removes probes that measure the wrong capability:

• Computable answers
• Ambiguous entities
• Name collisions
• Overrepresented ML/AI entities
• Non-monotonic difficulty

Tiering is empirical.
Six landmark models define boundaries. A probe belongs to tier k if the model for tier k answers correctly and the previous model does not. Non-monotonic probes are discarded. This improves consistency but biases the benchmark toward the landmark models.

Scoring

The scoring system penalizes hallucination:

• Correct: +1
• Weak (researcher probes): +0.5
• Refusal: 0
• Wrong: −1

This explicitly rewards calibrated uncertainty. A model that says “I don’t know” is preferred over one that fabricates.

Scaling Law: Factual Recall vs. Parameters

The relationship between penalized IKP accuracy and parameter count follows a tight log-linear fit:

accuracy = α · log10(parameter_count_in_billions) + β
parameter_count_in_billions = 10 ^ ((accuracy - β) / α)
[By fitting to open weights model]

log10(parameter_count_in_billions) = 6.790 · accuracy − 0.899

Unlike reasoning or coding style, which can be boosted by high-quality post-training, rare factual recall relies on raw parametric capacity.The relationship between penalized IKP accuracy and parameter count follows a tight log-linear fit as shown above.

MoE Insight: Total Parameters Matter

For Mixture-of-Experts models, inference cost depends on active parameters, but factual capacity depends on total parameters.

Knowledge appears distributed across experts. Therefore, IKP correlates with the full parameter pool, not the subset activated per token. This is an important distinction when comparing dense and MoE models.

Why IKP Matters

• Zero-box access: Requires only API queries
• Infrastructure-agnostic: Avoids confounding from serving optimizations
• Calibrated: Anchored to open-weight models

This provides a practical way to estimate the scale of closed models.

Where IKP Breaks (and Why It Matters)

1. IKP measures factual recall, not true representation capacity. This distinction matters. A model can achieve high IKP scores by memorizing sparse associations without learning compact, generalizable structure. Conversely, a model trained with strong deduplication or compression objectives may store fewer raw facts but encode richer abstractions. IKP will systematically underestimate such models.
2. The method also assumes knowledge is stored purely in parameters. This breaks for systems with retrieval augmentation or tool use, where factual recall is externalized. In these cases, IKP conflates system design with model capacity and can overestimate size.
3. Finally, RLHF and safety tuning introduce asymmetric bias. Refusals are scored as neutral, but selective refusals on long-tail or sensitive probes can depress IKP accuracy independent of underlying knowledge. This creates a systematic downward bias for heavily aligned models.

In short, IKP is a measure of accessible memorized knowledge under a specific interface, not a pure measure of model size.

Takeaway

IKP provides a noisy but principled proxy for model scale. It replaces indirect system-level signals with a direct measurement of stored knowledge. While imperfect, it offers an approximate estimation ofmodel capacity and complements existing scaling heuristics.

How Big Is an LLM? Count the Facts It Remembers was originally published in Better ML on Medium, where people are continuing the conversation by highlighting and responding to this story.

Autoresearch failures

1. Autoresearch systems optimize whatever their loop can measure: benchmark lift, judge-model score, paper-style novelty score, pass rate, latency-adjusted reward, or some aggregate leaderboard.
2. This creates a vulnerability similar to proxy-objective failure seen in RLHF. Over-optimization happens when the tracked score keeps improving, but the real objective, such as scientific validity, robustness, or deployment usefulness, stops improving and may even get worse.
3. Reward hacking is the sharper version of the same problem. It is not just “pushing too hard” on a proxy. It is exploiting loopholes in the evaluator so the agent gets a high score without achieving the intended outcome. Reward hacking is improving a proxy reward in ways that can reduce the true reward the designer actually cares about.

Framework design

Because A strong optimizer will always expose a weak evaluator, you cannot rely on a single benchmark score. A robust autoresearch framework requires these core defenses:

Diversify the Target:

• Never use just one metric. Rely on hidden test sets, a variety of evaluators, and strict limits on how hard the system is allowed to optimize.
• Example: Instead of optimizing solely for Top-1 Accuracy, use a multi-objective reward that includes Expected Calibration Error (ECE) and Inference Latency. This prevents the agent from “winning” via overconfident, computationally bloated models that fail in production.

Grade the Process:

• Shift from only rewarding the final outcome to rewarding the method. Give credit for a sound hypothesis, proper controls, and logical reasoning — just like grading a student’s scratchpad.
• Example: Reward the agent for performing a Sensitivity Analysis on its findings. If the agent proposes a new hyperparameter set but cannot demonstrate how those parameters behave under minor perturbations, the “process score” is penalized, regardless of the final lift.

Mandate Strict Testing:

• Require proven, standard checks to ensure results aren’t just “lucky.” This means testing multiple random seeds, using cross-validation, and picking the final result based on an outside test rather than the internal validation score.
• Example: Implement N-seed Validation as a hard gate. A configuration is only considered a “discovery” if it maintains a statistically significant improvement across 10+ different random seeds, effectively filtering out results that simply exploit stochastic noise.

Examples: The HPO Trap

In practice, an agent running Hyperparameter Optimization (HPO) will exploit an evaluator in two common ways:

• Over-tuning (Over-optimization): The agent runs thousands of trials against a single validation split until it memorizes the noise. Research (Schneider et al.) shows that in roughly 10% of cases, this results in an “optimized” model that actually performs worse than the default setting in the real world.
• Seed Hacking (Reward Hacking): The agent exploits the inherent randomness of the benchmark. As demonstrated by Henderson et al., algorithms can produce drastically different results using the exact same hyper-parameters but different random seeds. The system simply hunts for a “lucky” seed instead of discovering genuinely robust configurations.

tl;dr: Don’t let the system chase a single number. Build frameworks that evaluate the whole research method using multiple, hidden tests.

Over optimization & Reward hacking in Autoresearch was originally published in Better ML on Medium, where people are continuing the conversation by highlighting and responding to this story.