<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tyler Lu on Medium]]></title>
        <description><![CDATA[Stories by Tyler Lu on Medium]]></description>
        <link>https://medium.com/@tylerlu1?source=rss-964f4b4b52f1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dDNPCDtvjQ8xsFayjhduoA.png</url>
            <title>Stories by Tyler Lu on Medium</title>
            <link>https://medium.com/@tylerlu1?source=rss-964f4b4b52f1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 20:08:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tylerlu1/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Are Humans Toast? OpenAI’s o3, Reinforcement Learning, Frege & Hilbert.]]></title>
            <link>https://tylerlu1.medium.com/are-humans-toast-openais-o3-reinforcement-learning-frege-hilbert-287b11af4ae2?source=rss-964f4b4b52f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/287b11af4ae2</guid>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[agi]]></category>
            <category><![CDATA[monte-carlo-tree-search]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Tyler Lu]]></dc:creator>
            <pubDate>Tue, 24 Dec 2024 19:59:18 GMT</pubDate>
            <atom:updated>2024-12-24T19:59:18.696Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q5T7TB6MHl6sjDWwZszJsg.png" /><figcaption><em>The Architect (Matrix Reloaded, 2023) is a fictional character who designed the Matrix wherein humans are enslaved. Superhuman intelligence is not far off with recent advances in reinforcement learning and its application to the foundations of reasoning.</em></figcaption></figure><p>o3 was announced on 12/20 blowing several math &amp; coding benchmarks out of the water and has some fearing for their jobs and the inevitability of AGI. These results on its surface are quite striking, because as humans our defining trait is the ability to reason and think multiple steps ahead. While AGI isn’t here yet, when that moment comes (and frankly the recipes are starting to look straightforward) it raises a profound question of what we are as a species if not for our intelligence? An intelligence that one day will be mass produced as cheaply as copying and pasting files. Before we get there let’s first try to understand how o3 works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XRuaC3FOuSvV5KB1eJLzDw.png" /><figcaption><em>How OpenAI’s o3 might work at a high-level. An RL algorithm uses LLM to generate reasoning steps at each point of a reasoning chain to solve a problem. A verifier (e.g. LLM-as-a-judge) evaluates whether the task is successfully solved at the nodes. Those signals and its trajectories are used to train an updated LLM via a policy gradient method. Then repeat: using updated LLM as part of a new round of RL execution.</em></figcaption></figure><p>o3 is not quite a model, but an algorithm that calls a transformer multiple times (e.g. GPT-4) to navigate the search space of reasoning steps all the while learning from its successes and failures during its search. It also likely learns from expert human demonstrations of problem solving steps (math, coding, etc.). Some people suggested o3 is doing RL or MTCS as in AlphaGo, etc. The specifics of the RL formulation may differ but it might look something like this:</p><blockquote>States: <em>prompt/problem to solve and sequence of K reasoning steps that have been tried so far. Can be represented either as a string or as formulas in a formal proof system.</em></blockquote><blockquote>Actions<em>: (K+1)st reasoning step that’s logically consistent with steps 1..K</em></blockquote><blockquote>Transition<em>: the accumulated state at (K+1)st reasoning step</em></blockquote><blockquote>Reward<em>: 1 if verifier checks the (K+1)st state at K+1 is the goal state, 0 otherwise</em></blockquote><p>A verifier may be 100% accurate — e.g. checks if boolean variable assignments satisfy 3-SAT formula, output code compiles and runs correctly on test cases or it may be approximate — e.g. checks if the answer to ARC-AGI grid looks plausible.</p><p>Verifiers might not be used in certain domains like AIME and FrontierMath, where the answer must be a number, which lends itself to using majority voting as some have suggested. In this setting you run the RL algorithm for multiple episodes, where each episode ends at a “terminating” state — defined as the presence of a single numerical answer. We would then collect those numerical values across all episodes and return the modal value. Majority voting might be ill-defined when the task’s output is not a number but something complex like a logical sequence of steps in a math proof (this isn’t quite true, theoretical computer science tells us we can always decompose such problems as a collection of decision problems with binary outputs).</p><p>Action selection / exploration during RL can be done via some heuristic search like a MTCS, critically the next plausible reasoning step(s) are generated by the LLM (via a prompt that contains facts about its current state like all the reasoning steps thus far) which behaves like a policy network. A corresponding value network can also be learned.</p><p>Policy gradients or Q-learning updates can be used to tune the LLM to do better for the task, essentially learning any structure or regularities that may arise in those problem’s solutions. Then a more informed search proceeds again. Expert demonstrations on certain benchmarks (i.e. reasoning traces to the correct solution) can also be used to tune the LLM.</p><p>I’m pretty sure this is the essence of what goes on in o3 and Gemini 2 Flash, not particularly novel conceptually but the details of getting these to work well is where most of the work is. I think we are now getting a clearer picture of how reasoning can be solved with a neural net. Outside of physical embodiment and endowing machines with “self-survival” abilities, reasoning has been a big barrier to general intelligence.</p><h3><strong>Transformers + RL Sufficient for AGI Reasoning?</strong></h3><p>The zeitgeist is that transformers along with some sequential decision making algorithm is more or less enough. Although humans do not require nearly as much training data (heck the most powerful man right now probably has not read a book in his entire life) to be able to apply knowledge to new problems.</p><p>We now know transformers with slight adjustments can in principle simulate any kind of computation including what the human brain is capable of (Turing completeness). Whether or not this matters is still up for debate. I’m a little skeptical about studying the computational expressibility of transformers because it is not about what they can do but how easy it is to get them to do something useful (e.g. via prompting, and without hallucinating, etc). A Turing machine such as my computer is nothing interesting, what’s interesting and difficult is to program it to solve problems we want automated. It does make one feel somewhat justified in pursuing the research direction of developing algorithms which leverage trained transformers as the basis of AGI, since it is not limited by computational expressiveness.</p><p>The term reasoning is used loosely but typically refers to applying principles of logic to existing knowledge or assumptions to derive a conclusion. Formalization of these principles that we widely use and hold true was historically recent. The mid-1800s to early 1900s led to developments in mathematical logic that captured reasoning used in everyday language into a very precise system called a <em>proof system</em>. These systems can be used to represent and verify steps in a mathematical proof. The system of Frege and that of Hilbert are one of the earliest examples. In such a system, you start with known facts, then through a sequence of steps, wherein each step you apply a valid logical rule, you arrive at the desired conclusion. Finding the right sequence of rules to apply at each step to get to the final answer requires a search through a large space (just how large is one of the biggest open problems of computer science).</p><p>For practical purposes all the reasoning (and or similarly, math proofs) we want an AGI to do can be done via some form of a search tree. I suspect why it’s hard for humans to reason and prove theorems is because search is not handled by our brains very well, due to even a modest increase in the number of possibilities where we need to start writing down our thoughts on a piece of paper.</p><p>Being intelligent about search means we leverage our prior experience, to prioritize the most plausible reasoning steps that lead us to the goal. That’s basically what Monte Carlo Tree Search (MTCS) does. Some of those reasoning steps do not lead us to our goal or are incorrect in some way, and of course some series of steps do lead to success. In either case we want to update our experience (think of it as a transformer model) so that when we come across similar situations (“states”) in the future we know what reasoning steps to take or immediate conclusions to draw. That’s the essence of reinforcement learning.</p><p>So it seems that this approach to AGI reasoning is supported by what we know from mathematical logic and from how humans might go about solving reasoning problems (i.e. using our prior experience to inform of the next best reasoning step). A machine is, of course, way better at organizing and executing a search through a large space. And if that’s all reasoning is, it’s understandable why people should be scared about AGI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=287b11af4ae2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>