Reasoning Models: How Machines Learned to Think First
How We Got Here: Episode 20 of ~26. We are walking the AI timeline one contribution at a time.
The first time I used DeepSeek R1, I asked it to explain a process I hadn’t been able to get ChatGPT to explain.
Then I watched the reasoning traces.
Watching those steps unfold was my first experience with reasoning like that. I could follow it working through the process I’d asked it to explain.
It blew my mind.
I still think about that experience when I see a “Thinking…” panel. Being able to follow the steps changed how the interaction felt to me. The model was giving me something I could spend time with before deciding what to make of its answer.
That’s the experience I want to understand in this episode. How did researchers make this kind of working useful to a model? And what can we learn from being able to read it?
There are people and experiments behind that first encounter with R1. Following their work makes me appreciate what I was seeing, and gives me a better way to ask questions about it.
Where we left the agent
Last week, we looked at MCP, the Model Context Protocol: a shared way for an agent’s application to connect to tools and data. Before that, in Episode 18, we followed the agent loop. The model requests an action, reads the result, and uses what came back to choose its next move.
I can place reasoning more easily once that loop is in view. Before the model calls a tool, it may work through what to ask for. After the result comes back, it may need to work through what to do next.
Getting the right information is only part of solving a problem. If a tool returns four prices and a total budget, there is still arithmetic to do before anyone can tell me whether I can afford the purchase.
Give it somewhere to work
If I’m working out a calculation on paper, I can leave a subtotal on the page and come back to it. I don’t have to hold every number in my head while doing the next operation. I would happily give myself that help.
A language model can use written intermediate results too. It generates tokens, the pieces of text that can be words, parts of words, or punctuation. What it has already written is available when it generates what comes next.
So even an ordinary, fast answer involves repeated computation. The model was never doing one pass through the network for the whole reply. Writing more gives it more opportunities to compute. If what it writes includes a useful intermediate result, the next part of the response has something useful to work from.
Let’s follow that with a small, made-up problem. These calculations are illustrations, not records of model runs.
Four notebooks cost $3 each. You pay with $20. How much change should you receive?
Four notebooks at $3 each cost $12.
$20 minus $12 leaves $8 in change.
The model now has “$12” in front of it while producing the subtraction. That number carries information from the multiplication into the next operation. If I ask it to give me only the final answer, I’ve asked it to skip putting that working on the page.
Maxwell Nye and his collaborators made this part of the training in “Show Your Work,” their November 2021 scratchpad paper. They trained models to write intermediate computation for tasks including addition and program execution.
November 2021. I like having that date attached to the idea, because a thinking mode can feel like something that appeared fully formed in a chat window. People were already teaching models to put their working on the page years earlier.
The scratchpad was part of what Nye’s team trained the model to produce.
Who taught it to show the steps?
You can train a model to write intermediate work, as Nye’s team did. You can also give an existing model a prompt that demonstrates what you want.
In January 2022, Jason Wei, Xuezhi Wang, Denny Zhou, and their collaborators tried showing the working in the prompt itself. In their chain-of-thought prompting paper, a demonstration included a question, reasoning toward an answer, and the answer. Then came a fresh question for the model to work through.
If you remember the translation examples from Episode 12, this is a related use of the prompt. Examples tell the model what kind of continuation we want. Here, the examples include how to work toward an answer.
That’s few-shot prompting: putting a small number of demonstrations inside the request. With sufficiently large language models, Wei’s team found improvements on arithmetic, commonsense, and symbolic reasoning tasks.
In a miniature version of that prompt, we could supply these made-up demonstrations:
Question: Two pens cost $4 each. You pay with $10. How much change?
Working: The pens cost $8. Subtract $8 from $10.
Answer: $2.
Question: Three tickets cost $2 each. You pay with $10. How much change?
Working: The tickets cost $6. Subtract $6 from $10.
Answer: $4.
Then we add the notebook question and leave its working and answer for the model to produce. The demonstrations establish a sequence: find the total cost, subtract it from the payment, give the change.
We put the examples in the request. We haven’t changed the model’s learned parameters, the numbers adjusted during training. The examples are available while it answers this question.
I would keep that whole prompt with the result. If I accidentally make the pens cost $9 in my demonstration, I’ve given the model bad working before it has written anything. Someone looking only at its answer won’t see the mistake I supplied.
Then came the phrase you’ve probably typed yourself:
“Let’s think step by step.”
Takeshi Kojima, working with Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa, studied zero-shot reasoning in May 2022. Their method elicited steps without supplying handcrafted worked demonstrations alongside the question. That absence of demonstrations is what zero-shot means here.
Calling this “somebody discovered the magic prompt” skips over what each team actually tested. Wei’s group showed worked examples. Kojima’s group investigated getting useful working without them. In both cases, the model’s weights stayed fixed while it answered.
Those five words are easy to pass along. Knowing who tested them, and what they tested them on, gives me a much better starting point for trying them myself.
One answer, or several?
Suppose we generate five solutions to the notebook problem. Three finish with 8, and two finish with 7.
One option is to choose the answer that appears most often. This spends computation on alternative attempts, instead of using the whole budget to make one attempt longer.
The March 2022 self-consistency paper sits between the two prompting papers. Xuezhi Wang and colleagues sampled reasoning paths and gathered their final answers, choosing the answer most consistently supported across those paths.
But what if there’s a good answer among the attempts and the vote doesn’t choose it? We can train another model to judge the proposed solutions. That judging model is called a verifier.
Before either of those prompting papers, Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, and their OpenAI colleagues had tried this with mathematical solutions. Their October 2021 paper describes generating multiple candidates and selecting the one ranked highest by a trained verifier.
Their team also introduced GSM8K, a collection of about 8,500 grade-school mathematics word problems. A model could fail on a short question even when none of its individual operations required advanced mathematics. The collection gave researchers shared problems on which to compare methods.
A vote counts how often an answer appears among the attempts. A verifier has learned to judge proposed solutions. Neither gets the answer from an independent source of truth: five attempts can share the same bad assumption, and a verifier can prefer the wrong solution. A majority for 7 would still give us the wrong change.
If you handed me a disappointing result from this system, I’d want to see the candidates before deciding to change the generator. Was a good answer in there somewhere? Because if the model found one and the selection method threw it away, asking the model to try harder may leave the part that failed untouched.
The candidates are part of the result I’d keep.
Getting the answer right can hide a mistake
Of course, a scratchpad can carry a mistake forward too. If the multiplication produces $14, the subtraction may start from $14. Or we could get this:
Four notebooks at $3 each cost $14.
Subtract that from $20, leaving $8.
The final answer is right. Both calculations are wrong.
If I’m checking only that the answer is “$8,” I have to give it credit. And I really don’t want to give that working credit. Four times three isn’t fourteen, and twenty minus fourteen isn’t eight. Two mistakes have managed to arrive at the number I expected.
Get Cordero Core’s stories in your inbox
Join Medium for free to get updates from this writer.
“Let’s Verify Step by Step,” Hunter Lightman and colleagues’ May 2023 paper, took on that problem. Vineet Kosaraju and Karl Cobbe appear in this author list too, connecting it to the earlier verifier work. This time, the team compared feedback on the final outcome with feedback on the steps used to reach it.
They called feedback on the result outcome supervision, and feedback on intermediate steps process supervision. In their experiments on challenging mathematics problems, step-level feedback improved their system’s performance.
The team released PRM800K: 800,000 step-level human feedback labels used in that work. People judged individual steps, and those judgments helped train the system to judge them.
Eight hundred thousand labels. I don’t want to read past that as though the dataset arrived with the answers already attached.
Imagine reviewing our notebook example one line at a time. You mark the multiplication wrong. The next line reaches the expected answer, but you have to reject that subtraction too. Skimming to the bottom would miss both mistakes. You need to understand the operation well enough to judge it, even with a confident explanation and the right number sitting below it.
Our arithmetic is easy to inspect. The paper studied challenging mathematics. I find myself thinking about the person who has to keep paying that much attention to somebody else’s working, because “the model checks its work” makes the whole thing sound so self-contained.
In this study, people helped teach the checker what to approve. They belong in the history alongside the researchers who designed the experiment. I would be missing a large part of how we got here if I told you about a model grading its own work and left out the people whose judgments helped make that possible.
It also connects to our ChatGPT episode, where we looked at reinforcement learning from human feedback. Reinforcement learning uses rewards to encourage behavior during training. For a reasoning task, the reward can concern a checkable outcome, such as an answer matching a known solution or code passing tests.
Remember backpropagation from Episode 3? It helps calculate gradients through the model. An optimizer uses those gradients to adjust the learned parameters, and reinforcement learning gives us a way to make rewards for generated behavior part of that training.
What gets rewarded matters. Rewarding a correct answer can encourage useful working without checking every sentence along the way. There isn’t necessarily a person grading each hidden step. If code passes a test, I still need to know what the test actually checked before deciding what to trust.
Training it, then giving it time
I have to keep two kinds of work separate here, or “give it more time” becomes a very slippery instruction.
During training, computation goes into changing the model’s learned parameters. Those numbers carry what it learned into later requests. That’s training-time compute.
After you ask a question, the trained model does work to answer it. That’s inference-time compute, also called test-time compute. It can spend more computation generating intermediate reasoning without updating its weights during your request.
This separation between training and work during a request predates reasoning assistants. In the 2016 AlphaGo paper, David Silver, Aja Huang, and their DeepMind colleagues combined trained neural networks with tree search. The networks helped select promising moves and evaluate positions; search explored possible continuations while choosing a move. The paper reported a five-game victory over European Go champion Fan Hui.
Tree search considers a move and the possible replies, then further moves branching out from those. There are far too many branches to follow them all in Go. The learned judgments helped guide where to look.
The board helps me picture where the work is happening. The networks had already been trained, but the system still did work exploring continuations while choosing a move. Learning didn’t eliminate the need to search.
A paragraph about a scientific result is harder to judge than a Go position with legal moves and a way to determine the winner. I wouldn’t carry the analogy that far. What carries over is the separation between work done during training and work done when a particular answer is needed.
For language models, an August 2024 investigation asked how to allocate that test-time computation. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar compared approaches including verifier-guided search and revising candidate answers on mathematics problems.
Their results depended on how difficult the problem was and which model they started with. Allocating computation accordingly could work better than treating every question the same way.
That’s a useful question to bring to my own evaluation: where should the extra work go? Another attempt and a revision of the first attempt give the system different opportunities. Snell’s team tested how to allocate that budget on mathematics problems.
The pause becomes a product
OpenAI introduced o1-preview on September 12, 2024. Its research announcement described reinforcement-learning training that encouraged productive chains of thought before an answer. The full o1 followed in December.
Now the waiting had become part of a product people could choose. After all those papers, I think that is a remarkable thing: you could ask a question in a chat window and give the model more room to work before it answered.
OpenAI’s announcement showed improvements with more training computation and with more computation while solving a problem. Both contributed. Giving someone more paper gives them room; teaching them a better method changes what they can do with it.
This is also where I’d be careful with a comparison. A fixed trained model can receive different runtime budgets. But if I compare two separately trained products called “fast” and “thinking,” their training and parameters may differ too. The names on the buttons aren’t enough to tell me what caused the improvement.
The model that drew me in
DeepSeek released R1 on January 20, 2025. Its technical report, submitted two days later, credits a team including Daya Guo, Dejian Yang, and Haowei Zhang.
I want to slow down on the names for a moment, because R1-Zero and R1 are easy to blend together when retelling this.
R1-Zero started with an already pretrained language model and applied reinforcement learning without a preliminary supervised fine-tuning stage. Supervised fine-tuning means additional training on examples of responses we want the model to produce.
The researchers rewarded correctness and an output format without prescribing a particular reasoning strategy. They reported checking and revisiting approaches, alongside poor readability and language mixing.
Table 3 of the report contains a moment that reminds me of what drew me into R1. An intermediate version of R1-Zero, working through an equation, writes:
Wait, wait. Wait. That’s an aha moment I can flag here.
It goes back to the equation and reevaluates its steps. The researchers call it an “aha moment” observed during training. I hear a person catching themselves in that interruption. What we can examine is the generated reconsideration and the training method; the phrase itself doesn’t tell us whether the model experienced a realization as a person would.
For me, the surprise is in what the researchers supplied. They hadn’t written out that particular strategy for imitation in a preliminary supervised stage. They rewarded outcomes and a format, and the generated behavior included going back over an approach.
I can be excited about that and still want the training setup next to the quote. The excerpt shows reconsideration. On its own, it doesn’t tell me how much that particular reconsideration helped. For that, I’d need a checked result and a comparison that separated out what changed.
R1 used a more elaborate pipeline. Initial “cold-start” examples supplied readable reasoning before reinforcement learning. Cold-start means examples used to get this stage started; the underlying language model was already pretrained. The project repository describes two supervised fine-tuning stages and two reinforcement-learning stages.
I appreciate that the readability problems are in the account too. A useful behavior can arrive in a response that is difficult for another person to use, and the team had more work to do on that.
The release also let other people run R1’s weights and inspect its generated reasoning.
What are we actually reading?
Back to that thinking panel. What exactly has the interface given us to read?
For o1, OpenAI said users would receive a model-generated summary. The raw chain of thought stayed hidden. So before I interpret what a panel tells me, I need to know whether I’m reading the generated reasoning or a summary of it.
And even a readable reasoning trace can leave out something that influenced the answer.
Changing cues in a prompt gave Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman a way to investigate this in 2023. They altered cues including the ordering patterns of multiple-choice answers and observed explanations that failed to mention those influences.
That’s what faithful means here: the explanation accurately represents factors that influenced the answer. I can check whether the arithmetic works. Finding out whether the explanation accounts for what actually influenced the model is a harder question.
Yanda Chen and colleagues extended the investigation to reasoning models in research described in April 2025, including Claude 3.7 Sonnet and DeepSeek-R1.
They supplied hints, examined whether the hints influenced answers, and checked whether the reasoning acknowledged using them. Often, an influential hint went unmentioned.
That finding came from an intervention: change something in the input and examine the effect. The researchers hadn’t recovered a complete internal transcript. Their experiments also leave questions about harder tasks, where extended reasoning may be more necessary.
This is what I have to bring back to that first experience with R1. Watching the trace was compelling. I could follow the working, and that made the interaction feel different. Reading an explanation can help me inspect an answer without showing me every influence that went into it. Knowing that doesn’t make the experience any less fascinating to me.
I don’t think I can solve that by refusing to read the working. It can help me inspect a calculation or understand a proposed method. I just have to remember that an explanation I can follow may still leave an influential hint out. I can see the value in reading it and feel the difficulty of knowing what to trust, both at once.
Before turning up the thinking budget
If we were looking at a model for your work together, I’d start by asking what a good answer would let you do.
For code, which behavior does the solution have to satisfy, and how will we check it? A scientific calculation needs its assumptions and units with the final number. Once we know what a good answer has to contain, we have a way to recognize success before choosing how much thinking time to buy.
We could try several runtime budgets on the same questions, keeping the trained model fixed where the system allows it. The results belong with the time and cost of producing them. I’d keep ordinary questions in there too. The difficult ones are interesting, but the ordinary ones are part of what we’ll actually ask the model to do.
If we add a verifier or a test runner, that change needs its own record. Executing a useful check might improve the result, and we need to know which part of the system did that work.
The unsuccessful attempts matter too. Was a valid solution generated and then lost during selection? Did the same arithmetic mistake survive several attempts? One impressive answer copied into a document won’t tell us. The original requests and outputs give us something to go back to together.
A wrong answer with more working is still wrong. But seeing the working can give us a place to investigate, especially if we know what the system was trained or configured to do with it.
I find that encouraging. There is something to do with each part of this history: try a worked example, look at the rejected candidates, find out what the checker learned to approve. The people behind these papers gave us those ways in.
That first experience with R1 still blows my mind. I want to keep watching the working unfold. And I have a question to carry into that next conversation: how much of my confidence comes from being able to check the steps, and how much comes from how familiar they sound?
Next Tuesday, we’ll follow the research that connected pictures with language, including the transformer’s move into images. That work was developing before o1. We’re returning to another branch of the history, with the same interest in how people made the pieces work together.
If you use reasoning models for research or software work, when has giving a model more time improved the result? I’m interested in what you could check afterward, especially when the explanation and the result left you with different impressions.
You can follow the series on Medium or subscribe on Substack. I’ll keep walking through the discoveries behind the tools we’re building with.
Cordero is a senior research software engineer at the Scientific Software Engineering Center at the University of Washington’s eScience Institute. Send this to the teammate whose answer to every model evaluation is to turn the thinking budget up another notch.
