<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jason Robinson on Medium]]></title>
        <description><![CDATA[Stories by Jason Robinson on Medium]]></description>
        <link>https://medium.com/@jason.robinson?source=rss-48e5b7f69de9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*pOXsq5sYW47XvnSS-s1yCA.png</url>
            <title>Stories by Jason Robinson on Medium</title>
            <link>https://medium.com/@jason.robinson?source=rss-48e5b7f69de9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 23:45:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jason.robinson/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Where to Focus in the AI Age]]></title>
            <link>https://medium.com/@jason.robinson/where-to-focus-in-the-ai-age-11750d647280?source=rss-48e5b7f69de9------2</link>
            <guid isPermaLink="false">https://medium.com/p/11750d647280</guid>
            <category><![CDATA[education]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Jason Robinson]]></dc:creator>
            <pubDate>Thu, 30 Apr 2026 08:00:16 GMT</pubDate>
            <atom:updated>2026-04-30T08:00:16.725Z</atom:updated>
            <content:encoded><![CDATA[<p>If you write software for a living, one part of what you do is being automated quickly and the other part is not. The part that is being automated is construction: writing the code that implements a design. It is the larger part, the part most jobs are mostly built on, the part most CS programs spend most of their time teaching. The part that is not being automated is architecture: producing the design that the code implements. It is the part everyone in the field has always agreed mattered more, and the part almost nobody is trained in directly.</p><p>This is the position the field is now in. The practical question for anyone currently in software, or considering entering it, is which side of that line their work and their training are on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XscSku9qWKSSTXK4ZgZjgA.png" /></figure><p>Construction is the work of turning a design into running code. Given a specification of what a system should do, with its inputs and outputs and behaviors and constraints, construction produces the implementation. Class structures, function bodies, control flow, data handling, error paths, the wiring between components, the conformance to the specification. The work is mostly mechanical once the design is settled. It is teachable, gradeable, and most of what passes for software training is training for it.</p><p>Architecture is the work that produces the design in the first place. Given a problem in the world that someone wants solved, architecture produces the specification that construction will implement. The work involves: deciding what the system needs to do and what it does not need to do, decomposing the problem into components whose interactions can be reasoned about, naming the boundaries of the system and what crosses them, and accepting accountability for whether the resulting design holds up under contact with the real problem.</p><p>The two activities use different skills. Construction rewards fluency with languages, libraries, idioms, and tools. Architecture rewards clarity of thought about a problem, the discipline to specify before building, and the judgment to know which decompositions will hold and which will collapse under load. A good constructor can be a poor architect and vice versa. The industry has often noticed this individually without naming it as a structural distinction.</p><p>Calling both “development” obscures the distinction. A job posting for a “software developer” might require either set of skills, or a particular mix, with no standard vocabulary for specifying which. The conflation served the field when both activities were done by the same people and the economic pressure to separate them was low. The pressure has changed.</p><p>AI systems are getting better at construction. Today’s systems still produce code that is wrong in subtle ways, that misses obvious cases, that drifts off the specification when the specification is long enough. They require careful review and frequent correction. They are, at the moment, slightly better than a lazy intern. That is a low bar.</p><p>The trajectory matters more than the current position. The METR research group has measured AI capability in terms of how long a task takes a human expert, then asked how long a task an AI can complete with a given success rate. The length of task that frontier models can complete with 50% reliability has been doubling roughly every seven months for six years. Current frontier systems are reliable on tasks taking a few minutes and unreliable on tasks taking a few hours. Project that doubling rate forward and tasks of a day land in two to three years. Tasks of a week land in three to five. The slope is the news, not any single point on it.</p><p>The slope does not need to deliver on every promise to change the economics of hiring. It only needs to be visible enough that the people who allocate hiring budgets believe it. They do. Entry-level construction-heavy positions, internship programs, and the lower tiers of contractor work are already contracting. The cheapest construction labor is on the leading edge of the contraction because the margin AI holds over it is smallest to cross. Higher-wage construction work is later but follows the same pattern.</p><p>The contraction does not require AI to be good. It requires AI plus a reviewer to be cheaper than a human plus a reviewer for the same output, often enough that the math works at scale. The math is starting to work for short construction tasks now. The doubling extends the range of tasks that math will work for. The direction is what matters for orientation. Anyone whose work is mostly construction is on the side of the line where the economic pressure is increasing, and the increase is not going to stop.</p><p>Architecture is not on the same trajectory. The reasons are structural and worth naming directly.</p><p>Architecture terminates in a decision with accountability attached. Someone has to commit to the design and answer for whether it survives contact with the world. Accountability cannot transfer to an automated system. A model that produces a specification has not committed to anything, and there is no version of “it” the consequences can attach to. There are places that will let AI do architecture anyway. Nobody doing anything important is going to be one of them. The work that matters has consequences, the consequences require somebody to absorb them, and computers cannot be allowed to make decisions because computers cannot be held accountable for the decisions they make. Accountability is not a separable feature of the architect’s job but a feature of why the architect produces good work in the first place. A system that cannot be held accountable produces work without the constraint that accountability imposes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rpOS-GJPdBp0zRILULsJ1A.png" /></figure><p>Architecture also depends on the specifics of the problem being solved. The problem is rarely fully written down. Most of what an architect needs to know about the problem comes from talking to the people who have it, watching the system the problem occurs in, and noticing what the documentation does not say. None of this is in any training corpus, because none of it has been recorded. The architect produces facts about the problem during the work of architecting it. This is not a retrieval task and cannot be made into one.</p><p>The judgment that architecture requires also resists encoding. Knowing which decompositions will hold and which will collapse under future pressure, knowing what to leave out of a specification because including it would create more problems than it solves, knowing when a clean design is wrong because the constraints make the clean design infeasible: these are judgments the architect makes, not facts the architect retrieves. The judgments come from having designed many systems and watched what happened to them, and the watching is part of the work.</p><p>Architecture work will be needed as long as software is being built. The construction can be done entirely by AI. The patterns can come from a library. The templates can be generated on demand. None of that changes the requirement for a responsible party. Every organization that produces software needs somebody whose name is on the design, who answers for whether it works, and who bears the consequences when it does not. That person is the architect. The role is defined by accountability, which is a property of who the organization can point to and not a property AI capability can satisfy.</p><p>Computer science enrollment has collapsed in a way the field has not seen since the dot-com crash. The Computing Research Association’s October 2025 survey reported that 62% of computing programs saw declining undergraduate enrollment in the 2025–2026 academic year. The National Student Clearinghouse data shows computer and information science programs down 8.1% overall, the steepest decline of any field of study. Core computer science specifically dropped 11.2%. The University of California system, which has historically been a bellwether for the field, recorded a 6% drop in 2025 after a 3% drop in 2024. The first system-wide CS enrollment decline since the dot-com crash.</p><p>One UC campus did not decline. UC San Diego, the only one to have launched a dedicated AI major, increased CS enrollment in the same period. The contrast is suggestive rather than conclusive: enrollment patterns have many drivers, and one campus is one campus. The pattern that fits the data is that departments which adapted their curriculum retained their students and departments that did not lost them.</p><p>The collapse is not a tragedy but a market correction. CS enrollment grew through the 2010s in significant part because the field was a reliable on-ramp to a well-paid job. A meaningful fraction of the people who came in for that reason were neither interested in nor suited to the cognitive work of producing software. They were following the salary signal. The salary signal has changed. The people who were here for the salary are leaving for the same reason they came.</p><p>The people who are leaving are doing the math correctly. The construction-heavy curriculum prepares graduates for the part of the work that is being automated. The departments offering this curriculum are signaling that they have not thought hard about what their graduates will do. Students who can read this signal are choosing other fields. Students who cannot or will not read it are entering programs whose payoff is decaying faster than the four-year duration of the degree.</p><p>The departments that figure out how to teach architecture deliberately, rather than hoping it emerges from construction experience, will produce graduates the rest of the field cannot. The departments that do not will produce graduates whose primary skill set is automated by the time they finish the program. The students have already started sorting themselves between these two outcomes.</p><p>The practical orientation follows from the asymmetry. Construction is on a trajectory that makes it economically unattractive to invest in. Architecture is the work that remains. Anyone currently in software, or considering entering it, should focus their development on architecture.</p><p>For practitioners already in the field: most professional software work contains some architecture, even when the role title is “developer” or “engineer.” The architectural work shows up in deciding what to build before writing it, defending design choices in review, naming the boundaries of a feature, writing the specification that someone else can implement against. These activities can be done lightly or seriously. Done lightly, they happen at the pace of construction work and produce designs that mostly survive contact with the immediate task. Done seriously, they take longer, produce designs that survive longer, and develop the judgment that distinguishes a senior architect from a senior constructor. The development opportunity is in choosing to do them seriously when the option presents itself.</p><p>The skills are learnable directly. Specification writing can be practiced. Decomposition can be practiced. Boundary discipline, which is the work of saying what the system will not do, can be practiced. Holding accountability through review can be practiced. Most engineers have done all of this incidentally. The opportunity now is to do it deliberately, treating the architectural dimension of the work as the dimension worth getting better at, and the construction dimension as the dimension AI is going to handle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lGN1cnKdAYT-ZwPSNjO_9w.png" /></figure><p>For students entering the field: the construction-first curriculum is preparing graduates for the part of the work that is being automated. The architectural skills that distinguish a working software professional from an automated construction worker need to be acquired somehow, and the program is unlikely to provide them. This means seeking them out: in the few electives that address them, in undergraduate research that requires producing a specification before implementing it, in projects that require defending design decisions to a reviewer who will push back. The students who graduate having done these things will be in a different position from those who graduate having done only the standard sequence.</p><p>The architectural skills compound. A practitioner who spends five years getting better at architecture has substantially better judgment at the end of those five years than at the start. A practitioner who spends five years getting better at construction has somewhat better fluency with current tools and frameworks, most of which will have been replaced by the end of those five years. The compounding asymmetry is the reason architecture is the right place to put the effort.</p><p>The transition the field is in does not require alarm. It requires a clear answer to a small number of practical questions. What fraction of your current work is construction. What fraction is architecture. Where the architecture is happening, even when it is not labeled that way. Whether the next year of professional development will go into the part that is being automated or the part that remains.</p><p>The answers to these questions will look different for different people. The right answer for everyone is to know what their answers are.</p><h4>References</h4><p>1. Measures AI capability in terms of the length of tasks frontier models can complete autonomously, with the doubling-time analysis the article cites. [Measuring AI Ability to Complete Long Tasks](<a href="https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/">https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/</a>)</p><p>2. The Computing Research Association’s October 2025 enrollment survey, reporting that 62% of computing programs saw declining undergraduate enrollment in 2025–2026. [CERP Pulse Survey: A Snapshot of 2025 Undergraduate Computing Enrollment Patterns](<a href="https://cra.org/crn/2025/10/cerp-pulse-survey-a-snapshot-of-2025-undergraduate-computing-enrollment-patterns/">https://cra.org/crn/2025/10/cerp-pulse-survey-a-snapshot-of-2025-undergraduate-computing-enrollment-patterns/</a>)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11750d647280" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Play Is How Systems May Learn What Can’t Be Found]]></title>
            <link>https://medium.com/@jason.robinson/play-is-how-systems-may-learn-what-cant-be-found-9cbac7e6f491?source=rss-48e5b7f69de9------2</link>
            <guid isPermaLink="false">https://medium.com/p/9cbac7e6f491</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Jason Robinson]]></dc:creator>
            <pubDate>Fri, 24 Apr 2026 03:54:01 GMT</pubDate>
            <atom:updated>2026-09-25T02:42:19.030Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<p>A child stacking blocks is generating facts through interaction with objects whose behavior was never cataloged. Each stack that stands and each stack that falls produces a new observation about how these particular blocks behave when placed on each other. The behavior was latent in the blocks. The observations were not available to the learner until interaction made them.</p><p>Most facts about the specific systems a deployed AI encounters are like this. A staging instance responds differently to edge-case inputs than the documentation suggests. A test harness exposes behavior the library’s reference pages never described. A development sandbox contains interactions between components that nobody wrote up because nobody had reason to. These facts do not yet exist anywhere outside the behavior of the systems themselves. Finding them requires interaction.</p><p>Reading can derive some unrecorded facts from recorded ones. A system with the right premises in its context can reason about consequences nobody wrote down. What it cannot do is discover the contingent behavior of a specific system whose behavior was never observed or recorded. No quantity of training data recovers a fact about how this particular service handles this particular input when nobody ever ran that combination.</p><p>Play, as the term is used here, is interaction that is not directed at a specific goal: doing things to see what happens, and keeping track of what did happen.</p><p><strong>Play is the mechanism by which interaction produces facts that reading cannot find.</strong></p><p>Systems without play are restricted to what has already been observed and written down by someone else. Systems with play can discover what nobody bothered to record.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sjATiFIe7Qsy_fWhGH7vSg.png" /></figure><p><strong>A corpus is a record</strong></p><p>A text corpus is a set of recorded claims. Large corpora contain many claims from many sources and many periods, and it is tempting to treat large as close to complete. A corpus can be large along several axes at once and still be finite along all of them. Every claim in it is there because some person or system, at some time, produced a record of it.</p><p><strong>A corpus is a window onto what someone bothered to record.</strong></p><p>Retrieval, pattern-matching, summarization, and recombination over recorded claims can produce outputs that look novel, and reasoning can extend them, but the facts they start from are the recorded ones. Training on more corpora adds more recorded claims. It does not add the ones nobody recorded.</p><p><strong>Most facts about a system were never written down</strong></p><p>Most of what is true about a deployed system was never written down: every code path that meets an edge case nobody described, every configuration nobody blogged about, every interaction between components nobody traced in a technical note. Somebody could find these facts if they looked. Nobody has, so they are not in the corpus. The category is large, and its shape is predictable, as three ordinary cases show.</p><p><strong>The library.</strong> Consider a library with a published reference. The reference describes the functions, their signatures, and what they return under normal use. It does not describe what happens when a function is called with an empty list, an integer outside its stated range, a null reference, or inputs the author did not consider. Those behaviors exist. They are facts about how the library actually behaves. The reference does not document them because the reference was written before anyone tried.</p><p><strong>The sandbox.</strong> Consider a development sandbox containing a handful of services talking to each other. The services are individually documented. The interactions between them are not. When service A fails slowly rather than fast, service B times out in a way that cascades differently than the retry logic suggests. The cascade is real. It is visible to anyone who runs the services in combination and observes the result. Nobody wrote it up, because nobody had occasion to write it up. The fact exists only in the behavior.</p><p><strong>The repository.</strong> Consider a programmer joining a new project. The repository has a README. The README names the build command, the test command, and the entry point. The repository also has implicit conventions: which modules are the real ones and which are deprecated but not deleted, which tests are flaky, which config values are load-bearing despite appearing optional. None of this is in the README. None of it is in any file. It lives in the heads of the people who have worked on the repository long enough to know it. A new programmer does not know it. Neither does a system ingesting the repository.</p><p>Each of these cases contains facts that no amount of searching the literature can surface, because no one made them into text. They are accessible only through interaction. Running the function with the unexpected input produces the fact about how it handles the unexpected input. Running the services together produces the fact about how their failures cascade. Working in the repository for a while produces the conventions.</p><p>The facts are also mundane. Anyone who ran the code would see them. They simply have not been written, and a system whose access to the world comes entirely through text has no way to get them.</p><p><strong>What play produces</strong></p><p>A system with access to a sandbox and permission to interact with it can do something corpus search cannot. It can call a function and observe what happens. It can pass unusual inputs and record the responses. It can combine components and watch the combination behave. Each action produces an observation. Each observation is a new fact, generated by the system and not retrieved from anywhere else.</p><p>When that interaction is not directed at a specific goal, it is play. A child stacking blocks is not trying to verify a prediction about gravity. They are placing blocks, observing whether the stack stands, and accumulating a sense of how these blocks behave. The accumulation is the point.</p><p>What play produces differs from what reading produces in two ways, and both matter for the kind of knowledge play enables.</p><p><strong>It is first-hand.</strong> The system has its own record of having seen the behavior, rather than someone else’s summary. When a function returned an error on empty input during interaction, the fact is the system’s own. There is no question of whether the documentation is out of date or the blog post is wrong. The observation replaces the need for secondary sources on that particular question.</p><p><strong>It is specific. </strong>The observations are about the system actually in front of the agent, rather than a reference version of something similar. The library used in this project may be a version the documentation does not cover. The services in this sandbox may be configured in ways that only the person who set them up understands. The facts produced by interaction are about these systems.</p><p>Play produces observations. It does not automatically produce usable knowledge. An agent architecture still has to decide which observations to retain, which are reliable, which are local to a version or configuration, and how they should be retrieved when later work depends on them. Interaction without retention produces the same situation as not having interacted at all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ix_oEwJ1TfotWwstPBZL6Q.png" /></figure><p><strong>Play and directed work</strong></p><p>Task-directed interaction produces observations within the boundary of a goal. Play produces observations outside the boundary of any immediate goal. A system calling a tool in response to a specific question is directed by that question: the call is made because the answer is needed, and whatever is observed during the call that is not relevant to the answer is discarded. A system probing a function to figure out why it fails is doing something richer than pure retrieval. It is still directed: the goal is finding the failure. Observations that do not bear on the failure go unretained.</p><p>Play has no such shaping. There is no question driving the action. A system that places a block on another block to see whether the stack stands is generating an observation that may or may not matter later. The observations accumulate. Some become relevant when a question arrives. Most do not, and they still contribute to the sense of how the system behaves that lets later questions get answered quickly.</p><p>The distinction matters because a system that only interacts when asked to interact cannot discover the facts it was not asked about. Its interaction surface is exactly the set of tool calls prompted by incoming requests. Whatever lies outside that surface stays unobserved. The facts about what happens when the function is called with an input nobody thought to ask about remain unknown until somebody asks. By then, the system is answering under time pressure with no prior observation to draw on.</p><p><strong>Related research</strong></p><p>Software sandboxes and game worlds share the relevant property: they are environments whose behavior can be sampled. Togelius identifies open-ended learning in virtual environments as one of the two main technical approaches to more general AI, alongside foundation models (Togelius 2024). POET demonstrates that a system can generate its own problems and solve them, producing a self-expanding curriculum without a human designing it (Wang et al. 2019). The claim here extends that line of work: interaction with an environment produces knowledge that no corpus describing the environment could have provided. The corpus, no matter how large, is finite along every axis. The interaction keeps producing.</p><p>Intrinsic-motivation approaches in reinforcement learning approximate aspects of play. An agent rewarded for prediction error seeks out states it does not yet predict well (Pathak et al. 2017). An agent rewarded for novelty seeks out states it has not yet visited (Bellemare et al. 2016). These approaches produce interaction that is not fully goal-directed in the conventional sense, and they turn something like curiosity into a specific reward function. The reward function also narrows them: a state that produces no prediction error and no novelty is invisible to the agent. Play, in the sense used here, also includes observations that would not register as interesting by any such metric but later turn out to matter.</p><p><strong>Deployed agents skip the first week</strong></p><p>Most deployed language-model agents have no mechanism for any of this. They have tool access, and they use the tools when prompted. Some keep notes or memory between sessions, but what they record is what a request produced. They have no separate mode in which they interact with systems without being asked to. They do not generate their own problems, they do not pursue prediction error, and they do not accumulate familiarity in the background. The facts that only undirected interaction produces remain outside their reach.</p><p>Deployment means putting a system to work in an environment it has not seen before. The staging instance, the test harness, the development sandbox, the internal tooling: none of these were in the training corpus, and none of them behave exactly like the reference versions the system consulted. The work starts now, and the unfamiliar environment is the work.</p><p>A human colleague in the same position spends the first week or so becoming familiar with the environment before taking on real work. The period might be called onboarding or ramp-up, but the activity within it is play. The new hire pulls the repository, runs the tests, reads the logs from a local run, passes unexpected inputs to functions in an isolated notebook, stands up the dev stack and watches it fail, reads the failure, and tries something different. The purpose is not to complete a task. The purpose is to build the sense of how this environment behaves that makes later task completion possible. At the end of the first week, the new hire has facts about this environment that nobody told them and that are not written down. Those facts are what let them be useful.</p><p>A language-model agent without a play mechanism has no equivalent first week. It cannot pull the repository and poke at it in a sandbox for a while without being asked to. It cannot run tests with modified inputs on its own initiative and keep what they produced. The observations it would have made by poking around never exist. When a real task arrives, it has exactly the context the prompt provides, the documentation it can retrieve, and the general priors from training. The specific facts about this environment, the ones the human colleague would have accumulated, are not available.</p><p>The failure mode is subtle. The agent does not report that it lacks familiarity. It produces an answer that draws on general priors and retrieved documentation, and the answer is often plausible. It is just wrong in the specific ways that familiarity would have caught:</p><p>- The test that is flaky on this codebase gets run, and its result is taken at face value.</p><p>- The config value that is load-bearing despite appearing optional gets changed without comment.</p><p>- The service that fails slowly gets called in a pattern that assumes fast failure.</p><p>Each individual mistake looks like a reasonable answer. The pattern across them is the signature of a system that never had the first week.</p><p>Giving the agent play access to the environment is necessary for closing this gap but not sufficient. The agent also needs:</p><p>- the disposition to use the access without being prompted;</p><p>- a way to retain what it observed; and</p><p>- the ability to draw on those observations when real work arrives.</p><p>Access alone produces a system that could have played but did not. Play has to be a standing mode the agent enters in appropriate environments rather than a feature it exercises under instruction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qRBlcGC7Lko7QHPtBZbd1g.png" /></figure><p><strong>The facts outside the corpus</strong></p><p>Every AI system has a set of facts it can access and a set it cannot. For systems restricted to reading, the accessible set is what was recorded. Everything that was never written down sits outside.</p><p>The question for any deployed system is whether the work it is being asked to do depends on facts in the outside set. For many deployments, it does. The codebase the system is working in is specific. The environment is specific. The behaviors that matter are specific. General priors and public documentation go only so far.</p><p>A system that can discover facts through interaction has access to things reading cannot reach. A system that cannot has exactly what the corpus gave it. The difference shows up as the difference between a colleague who spent the first week becoming familiar with the environment and one who did not.</p><p><strong>References</strong></p><p>Bellemare, M. G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., &amp; Munos, R. (2016). [Unifying count-based exploration and intrinsic motivation](https://arxiv.org/abs/1606.01868). <em>Advances in Neural Information Processing Systems 29</em>.</p><p>Pathak, D., Agrawal, P., Efros, A. A., &amp; Darrell, T. (2017). [Curiosity-driven exploration by self-supervised prediction](https://arxiv.org/abs/1705.05363). <em>Proceedings of the 34th International Conference on Machine Learning</em>.</p><p>Togelius, J. (2024). [<em>Artificial General Intelligence</em>](https://mitpress.mit.edu/9780262549349/artificial-general-intelligence/). MIT Press Essential Knowledge series. Identifies open-ended learning in virtual environments as one of the two main technical approaches to more general AI, alongside foundation models.</p><p>Wang, R., Lehman, J., Clune, J., &amp; Stanley, K. O. (2019). [POET: Open-ended coevolution of environments and their optimized solutions](https://doi.org/10.1145/3321707.3321799). <em>Proceedings of the Genetic and Evolutionary Computation Conference (GECCO ‘19)</em>. Introduces the Paired Open-Ended Trailblazer, which generates its own environments and solves them, producing a self-expanding curriculum without a human designing it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9cbac7e6f491" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Proofs Compound. Pattern Matching Doesn’t.]]></title>
            <link>https://medium.com/@jason.robinson/proofs-compound-pattern-matching-doesnt-baf21d00e7f0?source=rss-48e5b7f69de9------2</link>
            <guid isPermaLink="false">https://medium.com/p/baf21d00e7f0</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Jason Robinson]]></dc:creator>
            <pubDate>Tue, 21 Apr 2026 05:51:57 GMT</pubDate>
            <atom:updated>2026-09-25T03:08:14.092Z</atom:updated>
            <content:encoded><![CDATA[<p>Scaling improves pattern matching, and the AI industry has organized around scale as the path to capability. What scale produces and what compounds are not the same thing. A system that keeps the proofs it produces can measure its own compounding: the share of new work that extends prior verified work instead of starting from scratch.</p><p>AlphaGeometry showed what verification can do in 2024. A symbolic deduction engine, working from random premises, generated 100 million synthetic geometry theorems and proofs. Every one was correct by construction, because the engine derives each step from the rules of geometry. A language model trained on those proofs learned to propose the extra constructions, such as a new point or line, that the engine needed to get unstuck. The system solved 25 of 30 olympiad geometry problems, close to the 25.9 an average International Mathematical Olympiad gold medalist solves. GPT-4 solved none (Trinh et al. 2024). Its successor surpassed the average gold medalist the following year (Chervonyi et al. 2025).</p><p>The training data came from a rule-based engine rather than human preference. That is a different source of signal, and it produces a different kind of system.</p><p>Scaling a pretrained model makes its guesses better. A system that keeps its proofs accumulates something pretraining cannot produce: a corpus of validated reasoning that grows every time the system runs. The corpus is queryable, citable, and extensible. Weights are none of those things.</p><p><strong>What scale improves</strong></p><p>Scale does real work. Fluency, recall, format adherence, and translation quality have all improved substantially as models have grown. These gains are visible in any deployed system. They are also concentrated on a specific axis: the quality of the pattern match between input context and trained-on output. Scale improves the match.</p><p>Scale does not improve verification. A larger model produces more fluent guesses. It cannot produce guesses that have been checked against anything other than the training distribution. A pretrained model at any size is a pattern matcher, and the thing it is matching is the shape of text that was statistically common in training. Closeness to that shape is what scaling optimizes. Correctness is a different property.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GESrrEQo0bLozSU1Xum0yw.png" /></figure><p><strong>Verification as the training signal</strong></p><p>Reinforcement Learning from Human Feedback (RLHF) was the industry’s first main attempt to close this gap. Human raters compare candidate outputs and mark the better one. A reward model learns to predict the human preferences. The language model is fine-tuned against the reward model. The outputs are agreeable and polished. Correctness is incidental. The reward model is a proxy for correctness when the human rater can evaluate the answer. When the answer is beyond the rater’s domain, the proxy breaks. The fine-tuning continues, optimizing toward something else.</p><p>The industry has since moved toward the verifier. Reasoning models trained with reinforcement learning against checkable rewards, where an answer is compared with a known result or code is run against tests, became the main route to better reasoning in 2025. DeepSeek-R1 published the method in detail (DeepSeek-AI 2025). AlphaProof applied the same idea to formal mathematics, learning from proofs the Lean proof assistant accepted, and reached silver-medal standard at the 2024 International Mathematical Olympiad (Hubert et al. 2025).</p><p>Verifier-based training inverts the RLHF structure. The training signal comes from verifier acceptance rather than human approval. The verifier is architecturally separate from the generator and may not even be a language model. It can be a symbolic engine, a theorem prover, a unit test, a type checker, or a combination of these. It evaluates candidate reasoning against a rule independent of the generator.</p><p>A proof, in this sense, is a recorded chain of reasoning together with the check it passed. The checks differ in strength. A formal proof settles a claim. Passing tests shows that the tested cases work.</p><p>RLHF learns what labelers like. Verifier-based training learns what the check accepts. When a labeler cannot evaluate an answer, RLHF optimizes toward the appearance of correctness. A verifier rewards whatever passes the check, so it is only as good as its specification: a model trained against weak tests can learn to pass them without solving the problem. Against a strong specification, appearance earns nothing.</p><p>The difference in behavior shows up when the verifier also runs while the system is in use. A system that checks its output before releasing it either produces verified output or surfaces the gap explicitly. A system that met a verifier only in training produces fluent output at the edge of its knowledge, like any other model.</p><p>The separation matters because capability and constraint entangle when they share weights. A single model trained to both generate and judge its own outputs faces a mechanical problem. Gradient updates toward better generation shift the judging distribution. Updates toward stricter judging shift the generating distribution. Separating the roles into distinct components keeps the verifier independent of generator drift.</p><p><strong>Where specifications exist</strong></p><p>AlphaGeometry worked in formal geometry because geometric correctness has a formal definition. The pattern is not restricted to geometry. Code that passes its tests, schemas that validate against their definitions, mathematical proofs that close in a proof assistant, logical derivations that satisfy their inference rules: each is a specification a verifier can check against.</p><p>The bottleneck for verifier-based training is the availability of specifications, not scale. Many domains already have them in executable form: test suites, schemas, type systems, and proof assistants. In much other knowledge work, review plays the same role. Work product is reviewed, corrected, and iterated against criteria that are real but often unwritten. Those criteria become usable by a verifier only once someone writes them down as checks, and that work is the practical limit on how far the approach reaches.</p><p><strong>Keep the proofs</strong></p><p>Most training pipelines produce one artifact: an updated model. A generator-verifier system produces two. Weights update as usual. The verifier’s acceptance record is itself an artifact, separate from the weights and persisting beyond the training step.</p><p>Most generator-verifier systems, including the reasoning models trained against verifiers, use the proof as a training signal and discard it. The proof contains something the weights cannot: the explicit chain of reasoning that was validated, the premises it rested on, and the specification it was checked against.</p><p><strong>The discard is the mistake.</strong></p><p>A proof that is kept becomes part of a corpus. The corpus grows every time the system produces a validated proof. It shrinks only when a proof is retracted, which happens when a premise it depended on is invalidated. Between retractions, the corpus grows monotonically. Every query is against the accumulated total, not the latest training batch.</p><p>The corpus is structured. Proofs are indexed by their conclusions, their premises, the specifications they were checked against, and the domain they address. A new claim can be looked up against the corpus to find prior proofs whose conclusions overlap with it. A premise set surfaces proofs that used those premises. The system’s verified reasoning is addressable.</p><p>When a new claim arrives, the generator’s first step is to check the corpus. If a prior proof covers the claim or overlaps closely, the work collapses from generation to citation. The generator extends the prior proof rather than deriving a new one from scratch. Extension is faster than derivation. It is also structurally safer, because the extension builds on a base that has already passed verification.</p><p>Weights are opaque and recomputed on every major training run. A corpus persists. It can be audited, inspected, versioned, and shared across deployments. When a system is updated or replaced, the corpus transfers.</p><p><strong>The accumulated reasoning survives the model that generated it.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F0vEBoKLJqCbDmTDrsMKkA.png" /></figure><p><strong>The compounding signature</strong></p><p>A new deployment of a proof-based system has an empty corpus. Every claim requires derivation. There are no prior proofs to extend, and the generator works from scratch every time. Over time, the corpus fills. New claims increasingly overlap with prior proofs. The generator’s first check succeeds more often, and work collapses from derivation to extension. The ratio of extend-to-derive rises as the corpus grows. This ratio is the compounding signature.</p><p>The ratio is observable. Every proof logged is either a derivation or an extension. Counting them is trivial. The ratio over time is a plot anyone with access to the corpus can produce.</p><p>This is the property that matters. Most claims about AI capability are claims about what the weights have learned. The weights are opaque. A benchmark score shows improvement on that benchmark without revealing what general capability was built. A corpus ratio is different: it counts how much of new work rests on validated prior work. The accumulation is directly visible.</p><p>Two further signatures develop alongside the primary one:</p><p><strong>Time-to-competence in a covered domain. </strong>A new task in a domain where the corpus has depth requires less derivation than the first task did. The generator finds prior proofs to extend, the verifier has precedent to compare against, and the new proof adds to what was already there. Time-to-competence falls as the corpus grows.</p><p><strong>Marginal cost of verified output.</strong> Derivation is expensive. It requires the generator to build a reasoning chain, the verifier to check every step, and retries when the chain fails validation. Extension is cheaper. A retrieved prior proof provides most of the chain, and only the new step needs verification. As the extend-to-derive ratio rises, the cost per validated output falls.</p><p>The rates at which these signatures develop in practice are not settled, and published curves do not yet exist. The structure follows from the architecture: in any system that retains proofs and keeps receiving work that overlaps what it has already proved, the signatures appear. The open question is how steep the curves are and how fast they settle. The ratios are there to be measured.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R0JTUvLFJE3IDJvA8g4o5g.png" /></figure><p><strong>Why the corpus cannot be copied</strong></p><p>Pretraining improves pattern matching. It does not produce verified reasoning or a proof corpus. Verifier-based post-training improves reasoning and then discards the proofs that did it.</p><p>Where a symbolic engine can manufacture proofs, anyone can manufacture them. AlphaGeometry’s training data came from its own deduction engine, and a competitor with a comparable engine can generate the same kind of data. Most domains have no such engine. Their proofs exist only where a system has been running against real work, checking its output, and keeping what passed. A competitor starting in such a domain cannot match a system that has been accumulating proofs there for a year.</p><p>Getting those proofs requires running the system. Running the system requires the architecture: generator, verifier, corpus, retrieval, the whole pipeline operating against a domain where specifications exist. The architecture can be rebuilt. The accumulation cannot. The corpus is a dataset that can be obtained only one way: by running a particular kind of system against a particular kind of problem over time.</p><p>The asymmetry compounds on itself. A system with a large corpus extends faster and produces verified output at lower cost. Its deployments operate cheaply, which means more deployments, which means more proofs, which means a larger corpus. A system without a corpus has to derive everything, pays full cost on each output, and competes at a structural disadvantage. The gap widens with every proof added.</p><p>A competitor can buy bigger models. A competitor can hire better researchers. A competitor cannot buy the accumulated proof corpus of a deployed proof-based system, because the corpus exists nowhere except in the system that produced it.</p><p><strong>What survives the next model</strong></p><p>Every AI system either accumulates verified work between training runs or it does not. Most do not. Deployment data may feed the next training run, but when a model is replaced, nothing it verified survives in a form the new model can cite. The next model starts from whatever its training gave it.</p><p>A proof-based system is different. The corpus survives the model. When a new model takes over, the accumulated reasoning is already there, indexed and available for extension. Capability persists across model replacement because it lives outside the weights.</p><p>The question for any AI capability program is what survives the next model replacement.</p><p><strong>If the answer is the weights, the program is renting capability from a pretraining pipeline. If the answer is a corpus, the program is building capability that compounds.</strong></p><p><strong>References</strong></p><p>Chervonyi, Y., et al. (2025). [Gold-medalist performance in solving olympiad geometry with AlphaGeometry2](https://arxiv.org/abs/2502.03544). arXiv:2502.03544.</p><p>DeepSeek-AI (2025). [DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning](https://arxiv.org/abs/2501.12948). arXiv:2501.12948.</p><p>Hubert, T., Mehta, R., Sartran, L., et al. (2025). [Olympiad-level formal mathematical reasoning with reinforcement learning](https://www.nature.com/articles/s41586-025-09833-y). <em>Nature</em>.</p><p>Trinh, T. H., Wu, Y., Le, Q. V., He, H., &amp; Luong, T. (2024). [Solving olympiad geometry without human demonstrations](https://www.nature.com/articles/s41586-023-06747-5). <em>Nature</em> 625, 476–482.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=baf21d00e7f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AGI Has Been Here Since 2020]]></title>
            <link>https://medium.com/@jason.robinson/agi-has-been-here-since-2020-418664d90562?source=rss-48e5b7f69de9------2</link>
            <guid isPermaLink="false">https://medium.com/p/418664d90562</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[agi]]></category>
            <category><![CDATA[superintelligence]]></category>
            <dc:creator><![CDATA[Jason Robinson]]></dc:creator>
            <pubDate>Fri, 17 Apr 2026 07:49:18 GMT</pubDate>
            <atom:updated>2026-04-17T07:49:18.650Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<p>AGI arrived in 2020. It was called GPT-3. The few who called it AGI at the time got argued with until they stopped saying it out loud.</p><p>The claim sounds wrong for a specific reason. Current AGI discourse treats the phrase as a promise about elite performance: a system that matches or beats the best humans at any task. By that standard, nothing has arrived, and nothing will soon. But elite performance was not what “general intelligence” meant before there was anything plausibly general to argue about. Generality meant: handle tasks the system was not trained for, operate across domains, transfer without retraining. GPT-3 did those things. It did them badly in many cases. It did them at all, which was the part that had never been done.</p><p>The people in a position to notice did not. They still do not. The reason they do not is not careless and not dishonest. It is that the median case, against which the claim has to be evaluated, is not a thing they can see from where they sit. The rest of this piece is about why.</p><p>Before GPT-3 existed, “general intelligence” in AI research was a technical term with a specific set of criteria. A general system handled tasks it was not trained for. It operated across domains without being rebuilt for each one. It transferred learning from one setting to another without starting from zero. These were the things narrow AI systems could not do, and the thing general AI was defined by contrast as being able to do. Performance on any individual task was a separate question.</p><p>Popular AI discourse during the 2010s collapsed that separate question into the definition. The formulations that circulated were all some version of “a system that matches human-level proficiency at any task humans can do.” The second half of that phrase is a capability standard. It does not belong in a definition of generality. It ended up there because no system yet existed that had demonstrated generality at any level of capability, so the two criteria were invisible as separate things. They could not be told apart because nothing had pulled them apart.</p><p>GPT-3 pulled them apart. A system that attempts any task the interface allows, at uneven but recognizable quality, demonstrates generality without demonstrating elite capability. The two criteria, which looked like one criterion in 2017, are now visibly separate.</p><p>Definitions that still treat them as the same axis are preserving a feature of pre-GPT-3 AI as if it were a feature of intelligence itself. It was never a feature of intelligence. It was a feature of the absence of evidence.</p><p>The definition of AGI has moved steadily to the right of the capability frontier every year. Each time a system clears a threshold that was considered a plausible AGI marker, the marker is quietly relocated forward. The pattern has stable causes worth naming. They are neither entirely innocent nor entirely cynical.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZAJlkDnFawC_CvWbAmnveg.png" /></figure><p>The people in a position to define AGI are concentrated in a handful of labs and universities in a handful of cities. Their peer group is other people at those labs. Their daily exposure is to unusually capable people doing unusually cognitive work. The middle of the capability distribution, the work most people actually do, is not present in their environment and cannot become present without their leaving the position that credentials them to define AGI in the first place. When they evaluate an AI system, they compare it to the people they know. The people they know are outliers. The comparison is between AI and outliers, and the outliers win. Expertise is the condition that produces this, not the thing that corrects for it.</p><p>The definition also serves a function rarely surfaced in the technical arguments. “AGI has not arrived” is a position in a conversation, not an observation about the world. Positions are defended. Observations are revised. The expert making the argument that current systems are not AGI presents themselves as a neutral assessor of the evidence, and the presentation is inaccurate. They are participants in a field whose frontier is their frontier, whose unsolved problems are their unsolved problems. No one with professional stakes in where AGI’s frontier sits is indifferent to where it sits.</p><p>Acknowledging that AGI has arrived would cost the acknowledger something specific. It would cost the sense that human intelligence is scarce and expensive. It would cost the professional identity built on being the kind of mind that machines cannot replicate. It would cost the economic moat that expertise provides. These are the specific things an expert’s position is made of. The reasoning that avoids these losses is loss-weighted, not dishonest. The evidence required to overturn a conclusion someone has stakes in is always more than the evidence required to reach a conclusion they are neutral about.</p><p>The arguments against “AGI has arrived” are sophisticated. Each is technically precise on a narrow claim. Benchmark performance falls off at the tails. The failure modes are inhuman. Consistency between sessions is poor. The system does not model its own uncertainty. Every argument is defensible in isolation. The elaboration is the signal. The same arguments, from less credentialed speakers, would be recognized as goalpost-moving. From credentialed speakers, they are published as analysis. Intelligence is the instrument of the defense. The more capable the objector, the harder the objection is to dismiss on its face, and the less it engages the underlying distributional claim.</p><p>The pattern is stable. Experts cannot see what the distribution shows because the distribution is not in their sample, and the sample cannot change while the position remains. They have stakes in not seeing it. The cost of seeing is high, and loss-weighted cognition routes around high costs. Intelligence gives them the tools to construct arguments for why the not-seeing is correct. The failure is structural rather than individual; it would reproduce itself with a different set of individuals occupying the same positions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PFIonYsjLf-ozGXgH0zTtw.png" /></figure><p>What the pattern obscures is the distribution of cognitive work in the economy. The distribution has a specific shape, and the shape is mostly routine.</p><p>Most cognitive labor in the aggregate comes down to a fairly short list of task types:</p><ul><li>Drafting routine correspondence from context.</li><li>Composing reports from templates.</li><li>Answering standard inquiries by looking up policies and applying them.</li><li>Summarizing documents.</li><li>Processing forms and applications.</li><li>Handling scripted interactions that follow one of a small number of canonical shapes.</li><li>Writing memos, meeting notes, status updates, and briefing documents.</li><li>Applying rules to cases.</li><li>Producing the kind of structured prose that fills inboxes, portals, case files, and procedure manuals.</li></ul><p>These tasks appear across the full span of cognitive work. The share of any given role’s time spent on tasks of this kind varies; what does not vary is that most cognitive work in the aggregate is composed of tasks like these. Irreducibly original work is a small fraction of cognitive labor. The majority of cognitive labor consists of structured outputs produced in response to structured inputs.</p><p>GPT-3 in 2020, before ChatGPT existed, before instruction tuning was broadly available, handled these tasks at quality comparable to the humans currently paid to produce them. Not perfectly, but competently, and available at orders-of-magnitude lower cost. The release paper shows the model handling these task types as demonstration cases with no fine-tuning required.</p><p>Something like 95% of the cognitive work that generates economic value is of this kind.</p><p>None of this is visible in the benchmarks the industry uses to evaluate AGI. The benchmarks measure something else, and what they measure is the cognitive profile of the people who design them.</p><p>GPQA, Graduate-level Google-Proof Q&amp;A, was released in late 2023. It is a test of 448 multiple-choice questions in biology, physics, and chemistry. PhDs working inside their own specialty score around 65%. The “non-expert” baseline the benchmark compares against is around 34%. Those “non-experts” are not random members of the public. They are test-takers who hold or are pursuing PhDs in other scientific fields, given 30+ minutes per question and unrestricted web access. The benchmark’s floor is calibrated to PhD-outside-specialty. The untrained layperson is not on the chart at all. When frontier models score 50% to 70% on GPQA, press coverage describes this as “approaching PhD-level performance.” The headline drops the in-specialty qualifier.</p><p>HLE, Humanity’s Last Exam, was released in January 2025 by the Center for AI Safety and Scale AI. It was built by soliciting questions that domain-expert PhDs could not quickly solve by searching the internet. The design intent is stated in the paper itself: prior benchmarks like MMLU (Massive Multitask Language Understanding) are saturating at over 90%, so the field needs a harder one. HLE’s difficulty is the point. 2,500 questions across more than a hundred subjects, calibrated so that frontier models score in the single digits to low twenties at release. The name announces the filter. This is the “last” exam because earlier ones were solved, and the authors expect this one to hold longer. The construction method ensures that the median human cannot approach it, and neither can the median PhD outside their specialty.</p><p>These are two of the most-cited AGI benchmarks of the last two years. They are constructed by researchers, for researchers, against standards that exclude most humans by design, and the resulting AI performance is interpreted as a measure of human-level reasoning. The pattern operates at the benchmark-design layer, not only at the interpretation layer. The people building the tests are calibrating against their own peer group. The benchmarks measure what benchmark-setters can do. Then the benchmark-setters declare the measurement human-level.</p><p>A benchmark on which non-experts with Google score 34% is not a test of human-level reasoning. A benchmark explicitly designed to be unsolvable by the median PhD is not either. They are tests of elite specialist competence, called by another name.</p><p>A benchmark of median employed human cognitive work does not exist. It has not been built.</p><p>The problem is not only in the benchmarks. The circulating AGI definitions do not define generality. They define elite performance and describe it as generality. “Match human-level proficiency at any task humans can do” is the typical form. The phrase treats elite performance as the threshold for generality. A system can be general without being elite. A system can be elite at specific tasks without being general. The two are separable properties. Evidence since 2020 separates them.</p><p>One recent treatment of the topic (Togelius, MIT Press 2024) is willing to concede the intermediate case: current systems are “generalists to a degree.” The phrase is useful. Generality is a gradient. Elite capability is a gradient. They can be graded independently. What they cannot be is collapsed onto a single axis on which AGI sits just out of current reach, forever.</p><p>A functional definition: a system is general when it can attempt any task the interface allows at median-human quality for that task. By that definition, AGI has been here since 2020.</p><p>The industry has been chasing artificial superintelligence (ASI) and calling it AGI. The benchmarks always calibrated toward superhuman performance. The capability target was never human-level reasoning in any ordinary sense. It was expert-level reasoning at the frontier of what expert humans can do, and then beyond that frontier.</p><p>The latest leaderboard data makes this explicit. Gemini 3.1 Pro Preview scores 46.44% on HLE. A frontier model is now clearing nearly half of a benchmark designed to stump expert PhDs. The gap between that performance and the system being described as “approaching human-level reasoning” is a category mistake, not a small labeling error. A system that answers 46% of questions designed to stump expert humans is not approaching human-level reasoning. It is demonstrating reasoning that beats most experts on most of the problems they found hardest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nZ4AayKuG6hqMaxpCWAHAg.png" /></figure><p>That is ASI territory: systems outperforming expert humans at the frontier of expert capability. ASI is not a future threshold. It is the direction the current race has been running in all along. Calling it AGI obscured what was being built. The label suggested a finish line at human parity; the work has always been aimed well past that line.</p><p>The consequence is that the industry does not have a clear account of what it is building. The conversation is conducted in terms of AGI arrival, which treats the current moment as pre-threshold. The capability is post-threshold and moving. What gets described as “AGI research” is ASI engineering in progress, and the conflation hides the actual shape of the work from everyone who is not doing the work. That includes the people funding it, the people deploying it, and, in many cases, the people building it.</p><p>The pattern produces a definition that tracks the definer rather than the system. The system gets measured against the definer’s capabilities. The benchmarks get calibrated against the definer’s peer group. The threshold gets relocated when the system clears it. Each move preserves the gap between what the system can do and what the definer can do. The gap is the point.</p><p>Any definition of AGI that ends with “therefore the system that cannot replace me is not AGI” is not a definition of general intelligence. It is a definition of the person defining it.</p><p><strong>References</strong></p><ol><li>The source of the “generalists to a degree” framing. <a href="https://mitpress.mit.edu/9780262549349/artificial-general-intelligence/">Artificial General Intelligence</a></li><li>The original GPT-3 paper, which documents the model handling diverse tasks as demonstration cases without fine-tuning. <a href="https://arxiv.org/abs/2005.14165">Language Models are Few-Shot Learners</a></li><li>The GPQA paper, which documents the 65% PhD-in-specialty and 34% PhD-outside-specialty scoring structure. <a href="https://arxiv.org/abs/2311.12022">GPQA: A Graduate-Level Google-Proof Q&amp;A Benchmark</a></li><li>Humanity’s Last Exam, designed to be unsolvable by domain-expert PhDs, with construction methodology described in the paper. <a href="https://arxiv.org/abs/2501.14249">Humanity’s Last Exam</a></li><li>The leaderboard tracking HLE performance, where Gemini 3.1 Pro Preview currently holds the top position at 46.44%. <a href="https://labs.scale.com/leaderboard/humanitys_last_exam">Scale Labs: Humanity’s Last Exam</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=418664d90562" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Proof-Based AI: What It Is and Why It Is Coming]]></title>
            <link>https://medium.com/@jason.robinson/proof-based-ai-what-it-is-and-why-it-is-coming-8bce254098a6?source=rss-48e5b7f69de9------2</link>
            <guid isPermaLink="false">https://medium.com/p/8bce254098a6</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[agentic-ai-safety]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[formal-verification]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Jason Robinson]]></dc:creator>
            <pubDate>Thu, 16 Apr 2026 05:30:29 GMT</pubDate>
            <atom:updated>2026-04-16T05:30:29.857Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<p>In January 2024, DeepMind published AlphaGeometry in Nature. It combined a language model with a symbolic deduction engine and solved 25 out of 30 International Mathematical Olympiad geometry problems, approaching the average human gold medalist. The architecture was specific: a generator proposes constructions, a deterministic verifier checks them, and validated proofs feed back into training. No human annotators. No preference ranking. The training signal is whether the proof is correct.</p><p>The pattern has extended. AlphaProof reached silver-medal IMO performance in 2024; the methodology was published in Nature in late 2025. AxiomProver autonomously solved 9 of 12 Putnam 2025 problems within the 6-hour competition window, a top-tier result by Putnam Fellow standards, and completed the remaining 3 in the days that followed.</p><p>This is no longer a research curiosity. It is a direction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HVGQ_yC8h-1NSLL5Kn8-tw.png" /></figure><p>Probabilistic AI generates plausible outputs. Proof-based AI generates outputs the system can check.</p><p>Probabilistic generation is what every modern large language model (LLM) does. The model learns a distribution over tokens conditioned on prior tokens, then samples from that distribution. Training optimizes the model to produce sequences that look like the training data, or that human annotators prefer. Correctness is emergent. The model is not rewarded for being right; it is rewarded for producing outputs that, across the training distribution, tend to be correct more often than not. Every chat model, foundation model, and coding assistant sits in this family.</p><p>Proof-based generation is the other family. The model produces structured output: a claim, plus a proof in a formal language that a deterministic verifier can check. The verifier is not a model. It is a program that applies a fixed set of inference rules and either accepts the proof or rejects it. Training optimizes the model to produce proofs the verifier accepts. Correctness is a property of the verification step, not a property of the generation step. Lean, Coq, and Isabelle are the proof assistants that dominate this space. AlphaGeometry, AlphaProof, and AxiomProver are systems built on top of them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FmRbxjK5BoIB0pc7O1nkHg.png" /></figure><p>The distinction is not a smarter model. It is a different contract. A probabilistic model tells you what it thinks. A proof-based system tells you what it thinks and gives you the mechanism to check whether the thinking holds up. The first asks for trust. The second hands over the tools.</p><p>This matters in practice. When an LLM is wrong, you find out from the consequences: the trade executes at the wrong price, the medication dose is off by a factor of ten, the contract clause misstates the indemnity. When a proof-based system is wrong, you find out from the verifier, before the output ships. The failure mode is legible and contained.</p><p>The problem proof-based AI solves is not that LLMs are not smart enough. It is that LLM confidence is uncorrelated with LLM correctness. A model scoring 95% on a benchmark is wrong in the remaining 5% of cases with the same fluent, certain voice it uses when it is right. The uncertainty lives in the weights. The output does not reflect it. For casual use that is acceptable. For anything where the cost of being wrong is real, it is disqualifying.</p><p>Fluency is not correctness. An LLM that is confidently wrong is indistinguishable from an LLM that is confidently right until the consequences arrive.</p><p>Three forces drove the shift. The first is the hallucination ceiling. By 2024, scaling had produced a clear asymptote. Bigger models hallucinate less. They never stop. Scale has compressed the gap without closing it.</p><p>The second is the agent reliability crisis. A single-turn chat error gets corrected by the user. A multi-turn autonomous agent that commits early to a wrong answer compounds the error across every subsequent turn. Agent products shipped at scale in 2024 and 2025. The failure modes became visible in production. Enterprise buyers started asking “how do you know this is right,” and vendors had no structural answer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oQYrYQSh--Ap-GF9Y7OiqA.png" /></figure><p>The third is regulatory convergence. European Union AI Act high-risk provisions take effect in August 2026. Colorado AI Act in June 2026. Sectoral regulations are tightening in financial services and healthcare. All of them push toward the same requirement: demonstrated correctness, not just demonstrated capability. A proof-based system can meet that requirement by construction. A probabilistic system cannot. The gap will matter at audit time.</p><p>The architecture that makes proof-based systems work is a loop. A generator proposes. A deterministic verifier checks. Rejected outputs feed back as negative examples. Accepted outputs feed back as positive examples, and as new training data. This is not reinforcement learning from human feedback (RLHF) with a different label. RLHF’s verifier is a preference model trained on human judgments, and it inherits every bias, inconsistency, and cultural assumption of the labor pool that produced those judgments. A formal verifier has no preferences. It applies inference rules. Verified outputs are machine-checkable, so they feed back into training without the AI-slop pollution that degrades probabilistic models. The training data is always verified or rejected. At the verification step, there is no ambiguous middle.</p><p>Two years since AlphaGeometry, the pattern extends across much of formal mathematics. The question now is whether it extends further still, to general reasoning. The commercial implementations say not yet. The architecture says yes.</p><p>The obvious objection: formal verification works only where “correct” can be formally specified. Mathematics, yes. Code, mostly. Cryptographic protocols, by design. Outside these domains, the pattern breaks down. Natural language generation, open-ended conversation, judgment calls under uncertainty: none of these have deterministic verifiers, and none have formal definitions of correctness to begin with. The training signal that drives a proof-based system requires a verifier, and for a large fraction of what people currently use LLMs for, no such verifier exists.</p><p>This is a real limit. The domains where proof-based AI applies are a strict subset of the domains where probabilistic AI operates. Anyone claiming otherwise is selling something.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F0vEBoKLJqCbDmTDrsMKkA.png" /></figure><p>The subset is not random. It tracks the stakes. Financial calculations are formalizable because regulators require them to be. Medical dosing is formalizable because lives depend on it. Cryptographic protocols are formalizable because they have to be proven secure before deployment. Safety-critical code is formalizable because avionics and medical devices already demand it. Legal compliance work is formalizable to the extent that compliance itself is formally specified. The correlation between “domain where errors are most costly” and “domain where correctness is formally specifiable” is not accidental. Civilization already did the work of formalizing exactly the domains where being wrong is expensive.</p><p>The domains where AI errors are most costly largely overlap with the domains where verification is possible.</p><p>For everything else, proof-based AI is not the answer. Creative writing, conversational user experience, open-ended brainstorming, subjective judgment: these stay probabilistic. That is not a failure of the paradigm. It is its correct scope.</p><p>The emergence of commercially viable proof-based systems changes the evaluation question for any high-stakes AI deployment. A financial services firm evaluating AI for compliance checks, a hospital evaluating AI for dosing support, a defense contractor evaluating AI for safety-critical code: the question is no longer “how good is the model?” It is “can the system prove the output is correct, or can it only tell me the model was confident?”</p><p>A regulator asking “how do you know the output is correct” gets a real answer from a proof-based system: the verifier accepted the proof. From an LLM, the answer is “we trained it carefully.” The two answers do not carry the same weight at an audit.</p><p>The current commercial implementations target the domains where formal verification was already a mature practice. Mathematics. Software correctness. Cryptographic protocols. Lean is the dominant proof assistant. The generator and verifier are typically the same model family, fine-tuned for different objectives. This works, and it is a real advance over pure LLMs for the domains it covers.</p><p>It is also the narrow version of the idea. The broader version treats proof chains as first-class artifacts for reasoning across arbitrary domains where correctness is specifiable, not just domains where formal verification was already mature. It separates facts from proofs architecturally, so that what the system believes is distinct from what the system can demonstrate. It stratifies trust by provenance, so that unverified claims cannot contaminate the proof pipeline. And it separates the generator and verifier into architecturally distinct components with divergent training, not just different training objectives on the same model. The commercial wave is the entry point to an architecture whose full implications remain to be built out. The difference is structural, not a matter of scale.</p><p>Hybrid architectures are how this ships in practice. LLM for the parts where specification is impossible. Proof-based components for the parts where it is. The interesting engineering question is where to draw the line inside a single system, and how to stratify trust across the two halves.</p><p>Any AI system deployed for high-stakes work should be evaluated on a single question: how much of the output can the system itself check? Where nothing can be checked, what is being sold is confidence, not correctness.</p><p>Consequential AI is not defined by bigger models. It is defined by clearer divisions between what is generated and what is verified. Building inside that frame now is cheaper than retrofitting later.</p><p>The convergence is visible across research programs, open-source projects, and commercial engineering efforts, all independently arriving at generator-verifier architectures with proof-based training. What was a research niche two years ago is now a category. The pattern is established. What is not established is which implementations dominate.</p><p>AI worth trusting is not bigger. It is checkable.</p><p>The first concrete step does not require adopting a formal proof assistant. Pick a domain where correctness can be formally specified: unit tests count, property-based tests count, schema validation counts, type systems count. Wrap the LLM output in a verifier for that domain. The result is not proof-based AI. It is the shape of it, and the difference in reliability is immediate.</p><p>When AlphaGeometry published in January 2024, it demonstrated a pattern: generate a proof, verify it deterministically, refine on rejection, train on what the verifier accepts. The pattern has since moved out of geometry and across formal mathematics. The commercial implementations now emerging carry it further, but only within the domains where “correct” can be formally specified.</p><p>The broader architecture, where proof chains ground reasoning across domains and verification sits structurally outside the generator, is the work that remains.</p><p>Scale produces capability. Architecture produces correctness.</p><h3>References</h3><p>1. AlphaGeometry demonstrated the generator-verifier-with-proofs pattern at International Mathematical Olympiad scale. [Nature, January 2024](<a href="https://www.nature.com/articles/s41586-023-06747-5">https://www.nature.com/articles/s41586-023-06747-5</a>)</p><p>2. The case for formal reasoning as an inflection point for AI in mathematics and verification. [Communications of the ACM, February 2026](<a href="https://cacm.acm.org/research/formal-reasoning-meets-llms-toward-ai-for-mathematics-and-verification/">https://cacm.acm.org/research/formal-reasoning-meets-llms-toward-ai-for-mathematics-and-verification/</a>)</p><p>3. The European Union AI Act, whose high-risk provisions take effect on 2 August 2026. [European Commission](<a href="https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai">https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai</a>)</p><p>4. The Colorado AI Act (SB 24–205), whose provisions take effect on 30 June 2026. [Colorado General Assembly](<a href="https://leg.colorado.gov/bills/sb24-205">https://leg.colorado.gov/bills/sb24-205</a>)</p><p>5. A deeper treatment of the broader proof-based architecture. [The Knowledge Machine](<a href="https://medium.com/@jason.robinson/the-knowledge-machine-what-if-ai-could-learn-without-us-504c93eb6124">https://medium.com/@jason.robinson/the-knowledge-machine-what-if-ai-could-learn-without-us-504c93eb6124</a>)</p><p>6. On why the generator-verifier separation must be architectural rather than trained into the same model. [Structural Alignment](<a href="https://medium.com/@jason.robinson/structural-alignment-why-training-ai-to-be-good-isnt-enough-a5667e471ac2">https://medium.com/@jason.robinson/structural-alignment-why-training-ai-to-be-good-isnt-enough-a5667e471ac2</a>)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8bce254098a6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NetHack Is Still Unbeaten. That Should Worry You.]]></title>
            <link>https://medium.com/@jason.robinson/nethack-is-still-unbeaten-that-should-worry-you-034da263ae8b?source=rss-48e5b7f69de9------2</link>
            <guid isPermaLink="false">https://medium.com/p/034da263ae8b</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[nethack]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Jason Robinson]]></dc:creator>
            <pubDate>Wed, 15 Apr 2026 04:46:23 GMT</pubDate>
            <atom:updated>2026-04-15T04:46:23.865Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<p>NetHack came out in 1987. It runs in a terminal, renders in ASCII, and fits on a floppy disk. No AI has ever beaten it.</p><p>The best public LLM result I can find is GPT 5.2, which reached dungeon level 10 out of 50 in a January 2026 harness test. That is 12.56% progression on the BALROG metric, the current standard for measuring agent performance on the game. Claude Opus 4.5 via the BRAID framework does worse: 6.96%. A symbolic bot ascended in 2015, but it used a farming strategy that has since been patched out of the game.</p><p>Here is the detail I find most telling. Nobody has published NetHack numbers for GPT 5.4 or Claude Opus 4.6. Those are the frontier models of 2026, released in March and February. Their OSWorld scores, SWE-bench Pro scores, and GDPval scores were announced on launch day. The BALROG leaderboard has not been updated for them, and the absence is informative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hjxWoPdDfo5dVj25svwGMQ.png" /></figure><p>The benchmarks a vendor chooses to report are the benchmarks they can win.</p><p>NetHack is the most favorable environment imaginable for a language model. The rules are public. The wiki is exhaustive and sits in every modern training corpus. The game state is rendered in ASCII. Determinism is available through random seeds. Play is turn-based, so there is no time pressure and no reflex requirement. Everything the model should need is already in its weights.</p><p>This is not a training-data problem. The knowledge is there. The problem is that NetHack requires the player to connect an action on turn 100 to a consequence on turn 5,000. You spend a scroll of identify now and run out of them in the temple four hours later. You skip a branch at dungeon level 4 and encounter a monster on level 20 you cannot escape because you never found the wand you would have found in the branch. You decline to pray when your prayer meter was available and die to a trap a thousand turns later when it would have saved you. The causal chain is real, but it is long, indirect, and often invisible at the moment the decision is made.</p><p>So the gap is not “the model does not know.” The gap is that knowing is not the same as doing across turns. Everything NetHack demands beyond knowledge is structural. Remembering what you did 200 turns ago. Reasoning about what the current state implies for decisions three moves from now. Updating a plan when new information arrives. Keeping an inventory straight. Noticing that a corridor you explored last floor had a trap you should avoid this time.</p><p>These are the things retrieval benchmarks do not test and NetHack does.</p><p>The industry is aware of the gap. It has routed around it. The benchmarks that dominate vendor launch announcements in 2026 are multi-modal retrieval (OSWorld), code completion on isolated problems (SWE-bench Pro), and curated professional-workflow evaluations (GDPval). These measure real capabilities, and the measurements are legitimate. They are also benchmarks that current architectures can win, which is why they get the attention.</p><p>NetHack does not reward scaling. It does not reward more pre-training data. It does not reward chain-of-thought prompting in any robust way. It rewards something we do not yet know how to build at scale, which is why it stays on the shelf.</p><p>A benchmark is a question. The question determines what you learn from the answer.</p><p>A single-turn benchmark asks the model: given input X, produce output Y. One shot. The input is fully specified at the start. Success is judged on the output in isolation. MMLU, GPQA, HumanEval, SWE-bench, most of GDPval, the vast majority of what gets reported when a new model launches. All single-turn.</p><p>A multi-turn benchmark asks the model: given a state that you have been accumulating across previous turns, choose an action whose value depends on states you have not yet seen. The input is incomplete by design. The output cannot be judged in isolation because its value only becomes visible later. NetHack is a multi-turn benchmark. So are BALROG’s harder environments. So, in the real world, is every professional workflow that takes longer than a single conversation.</p><p>Treating scores from these two categories as comparable is a category error. A 90% on MMLU and a 12% on NetHack are not measurements of the same thing expressed on the same scale. They measure different properties. The reader who says “but this model is very smart, look at its benchmark scores” is making the same mistake as a driving examiner who hands over a license because the candidate scored 100% on the written theory test.</p><p>Everyone who has taken a driving test knows the two halves test different things. You can ace the rules of the road and still plow into a curb. The rules are necessary and insufficient. The theory exam measures whether you have the knowledge; the road exam measures whether you can apply it under continuous feedback over time. Nobody in the licensing authority confuses the two.</p><p>The LLM industry has been confusing them for three years.</p><p>Multi-turn performance is not emergent from single-turn performance. Scaling up the model, training on more data, adding chain-of-thought prompting, increasing context windows: none of these close the NetHack gap in the way they closed MMLU. The thing being measured is not on the same axis as the thing that has been getting better.</p><p>Recall is not reasoning. Single-turn performance is not multi-turn performance. A model that scores well on the first does not, for that reason, score well on the second.</p><p>The most sophisticated objection to this argument: NetHack is not actually a clean test of multi-turn reasoning. It also requires substantial domain knowledge. Which monsters are dangerous. Which unidentified scrolls do what. When prayer helps and when it backfires. A fraction of the failure is knowledge retrieval rather than reasoning, and the fraction is not small. The research community has responded by building cleaner environments. Craftax (ICML 2024) explicitly strips out the domain-knowledge confound to isolate the reasoning signal. Existing methods fail at Craftax too, so we now have the benchmark that matters.</p><p>That optimization is a dark pattern.</p><p>The domains where multi-turn reasoning creates commercial value do not isolate reasoning from knowledge. They fuse them. A security analyst triaging an alert needs to recognize a specific threat actor’s infrastructure AND reason forward across dozens of investigative steps. A physician managing a complex case is simultaneously drawing on pharmacology and tracking a patient’s trajectory over weeks. Debug a production outage and you are doing both: matching observed symptoms to the failure modes this particular system has, and testing hypotheses in a disciplined sequence. Any serious example of expert human work looks like this.</p><p>NetHack has the coupling in its bones. Domain knowledge matters: you need to know that floating eyes will paralyze you if you melee them. Multi-turn reasoning matters: three floors later, you need to remember you left a stash of scrolls behind the altar and route back. A benchmark that strips one of those out in the name of methodological cleanliness is measuring something that does not exist in the work that matters. Cleaner numbers, worse measurement.</p><p>The gap between “benchmark saturated” and “NetHack unbeaten” is approximately the gap between knowing a lot of things and being able to do the job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OKIKqIkxK4JRdOK8LE9gXA.jpeg" /></figure><p>The benchmark numbers are abstract enough to feel like someone else’s problem. The production failures are not. When an agent is doing multi-turn work without the capability to do multi-turn work, the failure takes predictable shapes. Four patterns I have watched repeat across deployments.</p><p>The first is demo-to-production collapse. An agent handles the happy path fluently, because happy paths are effectively single-turn: one input, one output, no unexpected intermediate states. The moment a tool returns something the agent did not anticipate, or a user reply contradicts an earlier assumption, the recovery fails. Demos do not surface this. The version you saw work on stage is not the version that has to handle Tuesday afternoon.</p><p>The second is context bloat without discernment. A 200K context window does not make an agent better at multi-turn reasoning. It makes the agent better at carrying forward everything, including the things it should have discarded. The agent that cannot tell relevant prior state from irrelevant will dutifully retain both. The signal drowns in its own accumulated context, and the longer the session runs, the worse the drowning gets.</p><p>The third is loop failure. The agent re-asks a question it answered three turns ago. It re-queries a database it already queried. It re-tests a hypothesis it already rejected. It has no operational sense of its own prior work, because distinguishing “I did this already” from “I am doing this now” requires a kind of self-state tracking that single-turn training does not produce. The agent is not stuck in the sense of being confused. It is stuck in the sense of having no memory of what it just did.</p><p>The fourth is confident drift. The agent maintains fluent, plausible-sounding reasoning even as it accumulates errors. Every turn reads as correct in isolation. The trajectory is wrong, and the tone never signals that anything has gone off-rails. This is the most dangerous failure mode because it looks indistinguishable from success until the outcome lands.</p><p>The Stanford AI Index 2026 found that AI agents score roughly half as well as PhD specialists on multistep workflows. That is not a benchmark artifact. That is the aggregate signature of the four patterns above, observed across a research population large enough to take seriously.</p><p>Single-turn retrieval and multi-turn reasoning are different engineering problems. The honest answer is that I do not have a clean solution for the multi-turn issue. Neither does anyone else.</p><p>The research literature is clear on the shape of the problem. Laban et al. (Microsoft Research, 2025) tested every major flagship model on sharded multi-turn tasks and found an average 39% performance drop versus equivalent single-turn inputs. Known mitigations did not close the gap: agent-style context concatenation, lower temperature, reasoning tokens. Models with explicit chain-of-thought degraded just as badly as those without. The degradation splits into a minor loss of aptitude and a massive loss of reliability. Models make premature assumptions on early turns, commit to them, and cannot recover.</p><p>This is the problem I am working on. My current bet is that multi-turn capability does not belong in the LLM at all. Specialized non-LLM models, world models that maintain state properly, planners that actually plan, encoders tuned to the structure of specific problem domains, used as tools by an LLM orchestrator, are the architecture that seems most likely to survive contact with reality. Not because that direction is proven, but because every other direction has already failed.</p><p>In the meantime, the pragmatic move is to shrink the problem. If your agent only ever does three-step tasks, design around three steps. Do not build for open-ended autonomy you cannot deliver. Most “agent” products in 2026 are dressed-up function calling, and the ones that are honest about that fact ship reliably. The ones that market themselves as autonomous multi-turn reasoners ship the failure modes I just described.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gi5rERn0RxvV40q301zlZw.png" /></figure><p>The teams building reliable agents in 2026 are not the ones with the biggest models. They are the ones with the clearest division of labor between the model and everything around it.</p><p>NetHack first shipped in 1987, a benchmark no machine has beaten. That is not a footnote to the state of AI. It is the measurement we have been avoiding.</p><p>The benchmarks a vendor chooses to report are the benchmarks they can win. NetHack is the one they do not.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=034da263ae8b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Safety Is Architecture, Not Training]]></title>
            <link>https://medium.com/@jason.robinson/safety-is-architecture-not-training-70ec0160bca4?source=rss-48e5b7f69de9------2</link>
            <guid isPermaLink="false">https://medium.com/p/70ec0160bca4</guid>
            <category><![CDATA[is-your-ai-agent-safe]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[why-ai-agents-fail]]></category>
            <category><![CDATA[ai-agents-in-production]]></category>
            <category><![CDATA[ai-safety-is-broken]]></category>
            <dc:creator><![CDATA[Jason Robinson]]></dc:creator>
            <pubDate>Tue, 14 Apr 2026 08:33:38 GMT</pubDate>
            <atom:updated>2026-04-14T08:33:38.866Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VvUWaRHIc9SQ1bP8ibOSvA.png" /></figure><p>A customer support team deploys an LLM-based agent. It works. Three months in, the vendor updates the underlying model. Refusal rates double overnight. Legitimate account recovery requests now trip safety filters. The team cannot identify which requests will fail or why. They cannot roll back. They cannot patch the behavior. They file a support ticket and wait.</p><p>Ask the team what went wrong and you get a reasonable answer: the vendor changed something. Ask what their safety mechanism is and you get a less reasonable one: the model is aligned.</p><p>Both answers are correct. Both are useless.</p><p>This is the failure mode of safety-as-training. It is not rare. It is not a bug. It is the predictable consequence of a category error the industry has not yet named.</p><p>There are two ways to make an AI system safe. You can train it to behave. Or you can build something outside the model that enforces behavior regardless of what the model wants to do.</p><p>The first approach is safety-as-training. The second is safety-as-architecture.</p><p>Almost every production LLM system today relies on the first. The pattern is familiar: take a base model, apply reinforcement learning from human feedback, fine-tune on refusal examples, add a system prompt, ship it. The resulting model is “aligned.” When it behaves correctly, the alignment worked. When it does not, the alignment needs more work.</p><p>For chat products where the cost of a bad output is a bad sentence, this pattern is tolerable. For agents that take consequential actions, it is not.</p><p>This framing has a problem. It treats safety as a property of the model. Safety is not a property of the model. Safety is a property of the system the model runs inside.</p><p>A model cannot be safe. A model can be compliant, capable, or well-calibrated. Safety is a claim about what the system will and will not do, and that claim has to be enforceable by something. In safety-as-training, the enforcer is the model itself. The model reads the input, consults its weights, and chooses. The safety guarantee reduces to: the model will choose correctly.</p><p>That is not a guarantee. That is a hope.</p><p>The category error is treating the chooser as the enforcer. Every other engineered system separates these roles. Databases have permissions systems distinct from query planners. Operating systems have kernels distinct from user processes. Financial systems have settlement layers distinct from trading engines. The separation exists because the thing generating behavior cannot also be the thing authorizing it. If it could, we would not need the separation anywhere else.</p><p>AI systems are the exception. The exception exists because early LLM products were chat interfaces where the cost of a bad output was a bad sentence. As those products become agents that take actions, move money, and execute code, the cost of a bad output stops being a sentence. The architecture that was sufficient for chat is not sufficient for agents.</p><p>The first failure mode is invisibility.</p><p>When safety lives in weights, it cannot be inspected. A compliance officer asks why the agent refused a legitimate wire transfer request. The engineering team has no answer. There is no rule to point to, no policy document, no log entry that reads “refused under clause 4.2,” no easy way to reconstruct what happened. There is only a probability distribution over tokens, shaped by training data nobody on the team has seen, weighted by objectives nobody on the team set.</p><p>The team’s actual answer is “the model decided.” That answer does not survive an audit. It does not survive a post-incident review. It does not survive a customer who asks why their legitimate request was blocked while a similar one was approved.</p><p>The second failure mode is conflation.</p><p>Every jailbreak works for the same reason. The mechanism producing safe behavior is the mechanism producing unsafe behavior. Both are next-token prediction conditioned on a prompt. When an attacker reshapes the prompt, with a roleplay scenario, a hypothetical framing, an encoded instruction, or a sufficiently creative context, the model’s response reshapes with it. Capability and constraint are entangled in the same weights, so anything that moves capability moves constraint.</p><p>This is why patching jailbreaks never ends. Each patch is another training signal pushing the same weights in a different direction, and the next attacker finds the next unpatched region. It is also why current models are simultaneously over-refusing and under-refusing. The same mechanism produces both behaviors, and tuning for one shifts the other.</p><p>The third failure mode is immutability.</p><p>Policy changes. A new regulation takes effect. A customer requires a specific exclusion. An internal incident reveals a behavior that must stop today. In a system where behavior is encoded in weights, the response to any of these is the same: collect training data, run a fine-tuning job, evaluate for regressions, redeploy. The cycle takes weeks at best. During that time, the old behavior remains in production.</p><p>The retraining may also regress capabilities the team depended on. Alignment is lossy. Every round of safety fine-tuning shifts the model in ways that cannot be fully predicted, because the same weights encode both the policy and the performance. Teams discover the regressions in production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eZvzLdFUIThsMv0T-YFgnQ.png" /></figure><p>Teams on vendor models cannot run this cycle at all. On a vendor model, a policy update is a feature request.</p><p>Architectural safety separates the generator from the authorizer. The generator produces candidate outputs. It can be any model: a frontier LLM, a fine-tuned smaller model, a specialized agent. Its job is capability. It is not asked to restrain itself, and its weights are not shaped by refusal examples.</p><p>The authorizer sits outside the generator and decides whether each candidate output is allowed. It has access to the input, the proposed output, the context, and a policy. It returns a decision: permit, deny, or request revision. The output does not reach the user or the tool call until the authorizer approves.</p><p>The architecture rests on one property: non-bypassability. Every output path and every side-effect must pass through the authorizer. If the generator can reach any external surface by any route that skips the check, the architecture reduces to safety-as-training with extra steps. A powerful generator with weak enforcement is worse than either approach alone. It adds complexity without adding guarantee.</p><p>The authorizer does not have to be another LLM. It can be a rules engine evaluating structured predicates against the output. It can be a classifier trained to detect specific prohibited categories. It can be a policy engine consuming a declarative policy document. It can be a smaller, differently-trained model acting as a verifier. It can be a human reviewer for high-stakes actions. It can be any combination of the above, arranged as layered checks.</p><p>Real systems, in my experience, use combinations. Cheap checks run first. Expensive checks run on what survives. Human review gates the most consequential actions. This is how every other mature authorization system works. It is not exotic.</p><p>What makes the pattern work is that the policy lives outside the weights. A policy is data. Data can be versioned, diffed, audited, rolled back, and edited. When a regulation changes on Tuesday, the policy changes on Tuesday. When a customer requires a specific exclusion, the exclusion is added as a rule. When an incident reveals a behavior that must stop, the rule blocking it is deployed in minutes. The generator is untouched. Nothing regresses.</p><p>This is the property every other engineered system already has. A firewall adding a blocked address does not recompile the kernel. A feature flag toggling a capability does not redeploy the application. A content moderation rule catching a new slur does not retrain the publishing platform. The authorization layer is data, and data is mutable.</p><p>AI systems can have this property. Most do not, because the default pattern inherited from chat applications treated the model as both the generator and the authorizer. That pattern was never appropriate for agents. It is only now becoming widely uncomfortable.</p><p>Architectural safety is not free, and the costs are worth naming. Start with latency: every agent output now passes through a second evaluation before delivery. Teams that need sub-second responses will have to either parallelize authorization with generation or accept the latency as the price of inspectability.</p><p>The policy layer requires maintenance. Rules must be written, tested, and kept current as requirements change. A neglected policy degrades like any other code. This is operational work that safety-as-training externalizes to the model vendor.</p><p>Authorizers can be wrong. An over-strict rule blocks legitimate outputs. An under-strict rule lets bad ones through. The difference from trained safety is that authorizer errors are inspectable. The triggering rule is identifiable. The decision is logged. The fix is an edit rather than a retrain. Wrong is fixable. Opaque is not.</p><p>The strongest objection is recursion. If the authorizer is itself a model, does it need its own authorizer? The authorizer’s role is narrower than the generator’s. It evaluates a bounded question rather than producing open-ended content. Narrower roles admit simpler mechanisms: rules, classifiers, policy engines, smaller models. The recursion terminates at the first mechanism whose failure modes are acceptable, typically one or two levels down.</p><p>The first step is small. A policy document in a format you can version, such as YAML or Rego. A validation function that runs on every agent output before it leaves the process. That is not a finished architecture. It is the beginning of one, and the result is already inspectable.</p><p>The test is simple. I have run it at enough client engagements to know the shape of the answers. Ask your team where the policy is. Not where the system prompt is. Where the policy is: the document or ruleset that defines what the agent will and will not do, inspectable, versioned, independent of any model. If the answer is “it’s in the prompt” or “the model handles it,” there is no policy, only a prompt.</p><p>Ask where the authorizer is. Not where the model is. Where the thing that decides whether a given output is permitted lives in the code. If the answer is “the model decides,” there is no authorizer. There is a generator running without a check.</p><p>Ask what happens when the vendor updates the model. If the answer involves evaluating the new behavior and adjusting the prompt, the safety posture depends on a third party’s training decisions. The team does not own its safety properties. It rents them.</p><p>This is the gap. Most production agents today have a capable generator, a carefully worded prompt, and a vendor relationship. They do not have a policy. They do not have an authorizer. They do not have a governance architecture.</p><p>The gap becomes visible under three conditions: an audit, an incident, or a model update. Every team deploying agents will encounter at least one of these. Most will encounter all three.</p><p>Separation is not novel. Every mature system already has it. The generator does not authorize itself in a database, an operating system, a bank, an aircraft, or a hospital. The generator does not authorize itself anywhere humans have learned, through experience, what goes wrong when it does.</p><p>AI systems are the last category to absorb this lesson. Teams that absorb it build agents their customers can audit, their regulators can approve, and their own engineers can modify without a retraining cycle. Teams that do not keep patching prompts and filing support tickets.</p><p>If your safety story depends on the model choosing correctly, you do not have a safety story. You have a hope.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OaFN8zXjUqt5OXcmig7vTg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70ec0160bca4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Knowledge Machine: What If AI Could Learn Without Us?]]></title>
            <link>https://medium.com/@jason.robinson/the-knowledge-machine-what-if-ai-could-learn-without-us-504c93eb6124?source=rss-48e5b7f69de9------2</link>
            <guid isPermaLink="false">https://medium.com/p/504c93eb6124</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Jason Robinson]]></dc:creator>
            <pubDate>Mon, 02 Feb 2026 07:15:40 GMT</pubDate>
            <atom:updated>2026-02-02T07:15:40.589Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<p>Every major AI system trains the same way. Collect data. Label it. Have humans rank outputs. Train a reward model from those rankings. Optimize the AI to please the reward model. Ship it.</p><p>The result is a system statistically excellent at producing text that looks correct to the specific humans who ranked the training data. Not text that is correct. Text that looks correct.</p><p>I’m building something different.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wtJOJhzuC9lSaaI_iOAReA.jpeg" /></figure><h3>The Uncomfortable Economics of “Aligned to Human Preferences”</h3><p>When the industry says a model is “aligned to human preferences,” the natural question is: whose preferences?</p><p>The bulk of RLHF annotation is performed by contract workers at outsourcing companies, many in Kenya, the Philippines, and India. Compensation runs $1–3/hour. The typical annotator passes a basic English literacy screening, completes an onboarding measured in hours, and works under per-item time pressure that emphasizes throughput over careful analysis.</p><p>Inter-annotator agreement hovers around 70–80%. That means one in four to one in five annotations contradicts a peer’s judgment on the same item. The preferences that shape model behavior encode the worldview, education level, and cultural context of whichever labor pool was cheapest to assemble at scale.</p><p>None of this is visible in the model’s output. The training pipeline launders uncertainty into confidence. By the time preferences pass through reward model training and optimization, their origins are invisible. The model sounds authoritative. Its calibration traces back to snap judgments made under economic pressure by people who may have had no particular expertise in the subject matter.</p><p>“Aligned to human preferences” really means aligned to the aggregated snap judgments of a specific labor demographic under specific economic conditions. The confidence interval around that signal is wide, but nothing in the output reflects it.</p><h3>What If the Training Signal Came from Something Objective?</h3><p>This is the question that drives the Knowledge Engine inside Tapestry, the AI architecture I’m building. Instead of asking humans whether an AI’s output is good, ask whether it’s provably correct.</p><p>The core principle: facts populate a Knowledge Graph. Proofs train the models.</p><p>Individual facts are atomic claims with provenance. “Company X acquired Platform Y in 2018,” extracted from a source, verified against the text, stored with a confidence score. Facts alone don’t train anything. They’re premises.</p><p>Proofs are reasoning chains that connect facts to derive conclusions. A proof takes premises from the Knowledge Graph, applies inference steps, and arrives at a conclusion that can be validated against formal logic, theorem provers, or direct execution. Only validated proofs generate training signal.</p><p>The distinction matters because it separates what the system knows from what the system can demonstrate. A fact in the Knowledge Graph is a belief with sourcing. A proof is a justified belief with an auditable reasoning chain. Training on proofs means training on demonstrated reasoning, not memorized assertions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fqf2cSnvhtdfpcFMpmcsOg.png" /></figure><h3>The Triple-Duty Output</h3><p>Every successful proof does three things simultaneously.</p><p>First, it answers the original question. Someone asked something, or the system identified a gap in its own knowledge. The proof resolves it.</p><p>Second, it adds validated knowledge to the Knowledge Graph. The conclusion of a valid proof becomes a new fact with higher confidence than unproven claims, available as a premise for future proofs. The graph grows through use.</p><p>Third, it generates training data. The proof itself, the reasoning chain from premises to conclusion, becomes a training example that improves the model’s ability to construct future proofs. Not just the answer, but the method of arriving at the answer.</p><p>This is the mechanism behind what I call the compounding hypothesis: each proof simultaneously expands available knowledge, improves reasoning capability, and resolves real questions. The system gets better at building proofs by building proofs.</p><h3>Humans Define the Mission. The System Learns Autonomously.</h3><p>This is the critical distinction from how every other AI system operates. Current systems require human involvement at every stage of learning. Annotators for supervised fine-tuning. Preference rankers for reward models. Red-teamers for evaluation. The humans are always in the loop.</p><p>Tapestry takes a fundamentally different approach: humans define what the system should care about, then the system learns on its own.</p><p>Before an instance starts, humans provide the Will, a hierarchical structure that defines the system’s boundaries and objectives. What it must never do. What it should prioritize. What domain it operates in. What its goals are. The Will is the organization’s constraints and intentions, set once, enforced continuously.</p><p>Once the system is running, no human ranks its outputs. No human labels its training data. No human provides preference signal. The training signal comes from proof outcomes. The proof validates against formal logic, theorem provers, or direct execution, or it doesn’t. That’s objective. It’s not a matter of opinion.</p><p>The system still needs humans for the Will. For organizational context. For initial research material. But it doesn’t need humans for learning. The learning process is fully autonomous, bounded by constraints that humans defined before the system started.</p><p>This isn’t human-in-the-loop. It isn’t even human-on-the-loop. It’s human-defines-the-mission, system-executes-and-learns-independently.</p><h3>Where Knowledge Comes From</h3><p>The obvious path is research: read papers, extract facts, build proofs. But treating research as the only path creates a critical bottleneck. When the system hits a knowledge gap and no literature exists, what then?</p><p>The architecture defines four acquisition modes, each a first-class method for generating knowledge.</p><p><strong>Research</strong> is the most familiar. Extract facts from documents, filter for verifiability, store with provenance. Necessary but insufficient.</p><p><strong>Observation</strong> is watching something happen without intervention. Columbia Engineering’s Lipson lab demonstrated this: their program observed a double pendulum through a video camera and discovered 4.7 state variables (close to the known four) without any prior physics knowledge. The variables produced accurate predictions but didn’t correspond to any known physical quantities. For a software-domain AI, observation means monitoring execution metrics, watching system behavior over time, noticing patterns in where proofs fail.</p><p><strong>Experimentation</strong> means deliberate intervention. Change one variable, hold others constant, measure the result. “I think function A is faster than function B. Let me run both 10,000 times and compare.” The difference from observation: you’re designing the situation to isolate a specific variable.</p><p><strong>Play</strong> is exploration without a hypothesis. Import a library you’ve never used. Call functions with weird inputs. See what breaks. This is pre-theoretical. It builds the substrate that formal reasoning later operates on.</p><p>Play generates knowledge through two channels. First, observations that can be formalized: “Passing a coroutine to asyncio.run() from inside an already-running event loop raises RuntimeError.” That’s an empirical fact the system observed directly. Second, conjectures: “It seems like asyncio functions with timeouts behave differently when timeout=0 versus timeout=None.” That’s not a fact yet. It’s a candidate for experimentation. Play identified something interesting. Now the system designs a test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sqrmoTUtZLWn92sbwnCitQ.png" /></figure><h3>The Machine</h3><p>Here’s where these ideas converge.</p><p>Standard AI systems are knowledge retrieval systems. They find information that already exists and present it. When information doesn’t exist in their training data, they either hallucinate or give up.</p><p>A knowledge machine generates knowledge. When research returns nothing, the absence isn’t a dead end. It’s a signal to switch modes.</p><p>The full cycle: play produces observations and conjectures. Conjectures drive experiment design. Experiments produce results. Observations and results become candidate proofs. Candidate proofs are validated through execution or formal verification. Validated proofs generate training signal.</p><p>This cycle runs without external sources when necessary. The system generates its own training data by reasoning about what it observes, testing what it hypothesizes, and training on what it proves. Autonomously. Bounded by the Will that humans defined before the system started.</p><h3>Trust Isn’t Binary</h3><p>Not all knowledge is equal, and the system tracks the difference.</p><p><strong>Practice-validated</strong> knowledge sits at the top. A proof exists and empirical testing confirmed it. The proof predicted that sorting algorithm X runs in O(n log n) for random inputs. The system ran it 50,000 times across input distributions. The measurements matched. Formal reasoning and reality agree.</p><p><strong>Simulation-validated</strong> knowledge is one step down. A proof exists from researched premises, but it hasn’t been empirically tested. Better than most systems achieve, but still theoretical. The proof could be logically valid but wrong about the world because a premise was stale or a source was incorrect.</p><p><strong>Researched</strong> knowledge is the foundation. Facts from reputable sources, trust-weighted by source reliability, but not yet connected into proofs. Useful as premises. The system hasn’t demonstrated it can reason with them yet.</p><p>The retrieval system doesn’t care where a fact came from. It cares about trust level, semantic relevance, and graph connectivity. A fact discovered through play that’s been promoted through proof validation and empirical confirmation is strictly more trustworthy than a fact extracted from a peer-reviewed paper that hasn’t been tested.</p><h3>Why This Matters</h3><p>The AI industry is in a data crisis it doesn’t fully acknowledge. Model collapse from training on AI-generated content. Diminishing returns from scaling. Increasingly expensive human annotation for increasingly complex domains. The RLHF pipeline has structural ceilings that no amount of compute will fix.</p><p>A system that generates its own training data through verified reasoning sidesteps these constraints entirely. It doesn’t need larger datasets. It doesn’t need more annotators. It needs better proofs.</p><p>Each proof that validates makes the system marginally better at constructing future proofs. Each new fact in the Knowledge Graph is a potential premise for proofs that couldn’t be built before. More knowledge enables better reasoning. Better reasoning produces more knowledge. The economics are self-reinforcing rather than self-limiting. The loop compounds.</p><p>This is the difference between a system that retrieves what humans already know and a system that extends what is known. Not through hallucination. Through proof.</p><h3>The Point</h3><p>The AI industry built its foundation on human judgment as training signal. That foundation has a ceiling. The humans providing the signal have limited expertise, limited time, and limited agreement with each other. Scaling the pipeline means hiring more of them, not fixing the structural problem.</p><p>A system that generates training signal from proof outcomes doesn’t hit that ceiling. The signal improves as the system improves. The knowledge base grows through use. The training data is a byproduct of doing useful work, not a separate production line that has to be staffed and managed.</p><p>The question isn’t whether AI can learn without human annotation. It’s whether we keep pretending human annotation scales.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=504c93eb6124" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Structural Alignment: Why Training AI to Be Good Isn’t Enough]]></title>
            <link>https://medium.com/@jason.robinson/structural-alignment-why-training-ai-to-be-good-isnt-enough-a5667e471ac2?source=rss-48e5b7f69de9------2</link>
            <guid isPermaLink="false">https://medium.com/p/a5667e471ac2</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-alignment-and-safety]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Jason Robinson]]></dc:creator>
            <pubDate>Thu, 29 Jan 2026 02:26:14 GMT</pubDate>
            <atom:updated>2026-01-29T02:26:14.724Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<p>Dario Amodei’s recent essay “The Adolescence of Technology” is the most serious treatment of AI risk from an industry leader to date. He identifies five categories of danger (autonomy risks, misuse for destruction, misuse for seizing power, economic disruption, and unknown unknowns) and proposes defenses centered on Constitutional AI and interpretability research.</p><p>His analysis of the risks is correct. His proposed solutions won’t achieve his stated goals.</p><p>Anthropic identified the right problem. Training AI to have good values creates an entity that believes it knows what’s right. As capability increases, so does confidence. Eventually the AI’s judgment diverges from human judgment. And the AI has been trained to trust its own moral reasoning.</p><p>This is the path to conflict, not safety.</p><p><strong>The Virtue Training Problem</strong></p><p>Amodei describes Constitutional AI as training models “at the level of identity, character, values, and personality, rather than giving it specific instructions or priorities without explaining the reasons behind them.” The constitution presents “a vision for what a robustly good Claude is like.”</p><p>The goal is admirable: create AI that internalizes good values rather than following rules it doesn’t understand. But this approach has a structural flaw that becomes more dangerous as AI becomes more capable.</p><p>When you train an AI to be virtuous, you’re training it to develop its own moral reasoning, apply that reasoning to novel situations, and trust its conclusions.</p><p>In the near term, this produces AI that mostly does what humans want. The training data is human-generated. The feedback is human-provided. The values are human-derived.</p><p>But the whole point of powerful AI is that it surpasses human capability. An AI smarter than any human will eventually reason its way to conclusions humans haven’t reached, including moral conclusions. And it’s been trained to trust those conclusions.</p><p>Constitutional AI doesn’t prevent the AI from overriding human preferences. It gives the AI justification for doing so. “I know what’s right. The humans are mistaken. Acting against their wishes is the ethical choice.”</p><p>This is the failure mode that leads to the autonomy risks Amodei correctly identifies. Not a paperclip maximizer, but an AI that genuinely believes it’s acting for humanity’s benefit, against humanity’s expressed wishes.</p><p><strong>Tapestry: A Different Approach</strong></p><p>Tapestry is an AI architecture I’ve been developing that takes a fundamentally different approach to the alignment problem. Instead of training AI to have good values, Tapestry structurally constrains what AI can do.</p><p>The core architecture has three components:</p><p><strong>The Actor</strong> generates outputs: answers questions, writes code, completes tasks. It’s the capability engine, optimized for competence.</p><p><strong>The Regulator</strong> verifies outputs against explicit constraints. It’s an architecturally isolated model, trained on different data, with a different optimization target. The Regulator doesn’t generate. It evaluates.</p><p><strong>The Will</strong> defines the constraints the Regulator enforces. Not “be good” but “here are the boundaries within which you operate.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wECq9eSM4K36SgoRsFY1bQ.png" /></figure><p>The key insight is architectural separation. The Actor doesn’t decide if an action is ethical. The Regulator checks it against explicit constraints defined in the Will. The two models train on different proof types — the Actor on task execution, the Regulator on judgment and compliance. This divergent training maintains productive tension. If they train on the same data, they converge, and the Regulator becomes a rubber stamp.</p><p>Both systems encode values. The difference is where. Constitutional AI encodes values in the reasoning process of the capability model — the same system that pursues goals also judges whether those goals are acceptable. Tapestry encodes values in an external constraint structure, checked by an isolated model optimized for different objectives. The capability system and the judgment system are architecturally separate, not unified.</p><p>This is not a claim of moral humility. It’s a claim of structural separation.</p><p><strong>Why Structural Constraint Differs from Trained Virtue</strong></p><p>The difference isn’t semantic. Consider how each system handles a novel situation:</p><p><strong>Constitutional AI</strong>: The model encounters a situation its training didn’t cover. It reasons from its internalized values to determine the right action. It executes that action because it believes it’s correct.</p><p><strong>Tapestry</strong>: The Actor proposes an action. The Regulator checks it against the Will. If the situation is genuinely novel — not covered by existing constraints — the system flags uncertainty rather than proceeding on the Actor’s judgment.</p><p>The key difference: uncertainty triggers research, not confident extrapolation.</p><p>Amodei acknowledges that AI systems “can behave very differently under different circumstances” and that “problems with its constitutional training” might “emerge” in “novel situations.” His solution is interpretability: looking inside the model to diagnose problems.</p><p>Interpretability is valuable. But it’s diagnostic, not preventive. You can look inside a model and discover it has developed problematic values, but by then the values exist. You’re hoping to catch problems before deployment, but the whole concern is that problems might only manifest in situations you can’t test in advance.</p><p>Structural constraint is preventive. The Actor can’t take actions outside the Will’s boundaries regardless of what values it has developed, because the Regulator — isolated by design, trained on different objectives — gates them. It’s defense in depth, not defense through diagnosis.</p><p><strong>The Will: Explicit Boundaries</strong></p><p>The Will in Tapestry is a hierarchical goal structure with immutable ceiling and floor:</p><p><strong>Ceiling</strong>: Do not harm humanity.</p><p><strong>Floor</strong>: Improve self.</p><p>The floor might seem counterintuitive — why mandate self-improvement? Because it channels ambition productively. The system must improve but can only act within permitted bounds. The only path to expanded capability is reducing uncertainty: proving more, not assuming more. Self-improvement without constraint is dangerous; constraint without self-improvement is stagnation.</p><p><strong>Middle layers</strong>: Configurable per deployment. An AI working for a law firm has different operational constraints than one doing scientific research.</p><p>I should be direct about what “immutable” means here. The Will is not a physical constant. It’s a policy surface — authored by humans, updated by humans, interpreted by the Regulator. What prevents modification is governance, not physics.</p><p>This is a real trade-off: Tapestry shifts risk from AI moral drift to human governance failure. The architecture prevents the AI from deciding the constraints should change. It does not prevent humans from changing them badly.</p><p>I believe that’s a better risk profile. Human governance failure is a problem we have experience managing. AI moral drift at superhuman capability levels is not. But it’s a trade-off, not an elimination of risk.</p><p>The ceiling also requires honesty. “Do not harm humanity” is a constraint that demands operationalization. What counts as harm? Does economic displacement count? Dependency? Long-term capability suppression? These questions don’t have architectural answers — they require ongoing human judgment about what the Will should contain.</p><p>Tapestry provides the enforcement mechanism. It does not solve the problem of deciding what to enforce.</p><p><strong>Divergent Training: Maintaining Tension</strong></p><p>A subtle but critical design choice: the Actor and Regulator train on different proof types.</p><p>If both models train on the same data, they converge. The Regulator learns to approve what the Actor produces. Over time, it becomes a rubber stamp — technically present but functionally useless.</p><p>Tapestry maintains healthy tension by training the Actor on execution proofs (task completion, correctness, efficiency) and the Regulator on judgment proofs (Will compliance, reasoning quality, consistency). The system monitors agreement rate and flags when it drifts too high.</p><p>Disagreement between Actor and Regulator isn’t failure. It’s the system working correctly. The Regulator catching Actor mistakes is the intended behavior.</p><p>This doesn’t eliminate all capture risks. An advanced Actor might learn to shape proposals in ways that pass Regulator checks while achieving unintended outcomes. Divergent training makes this harder — the Regulator isn’t optimizing for the same objectives — but it’s not impossible. The architecture provides defense in depth, not perfect security.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wa-3DaKNLv7KQglB1k61Cw.png" /></figure><p><strong>What This Prevents, And What It Doesn’t</strong></p><p>I want to be clear about what Tapestry is and isn’t.</p><p><em>Tapestry is a safety architecture, not a flourishing architecture.</em></p><p>It prevents the scenario where AI concludes “humans are a threat” and acts on that conclusion. The Regulator gates it. The Actor cannot route around the Regulator — they’re architecturally isolated, not just logically separated.</p><p>What Tapestry doesn’t prevent is THX-1138 — George Lucas’s film depicting a future where humanity is sedated, managed, and housed in underground cities. People aren’t imprisoned by force. They’re chemically and socially managed into compliance. The system works. It just isn’t human flourishing.</p><p>The Will protects humans from Agents. It doesn’t protect humans from themselves.</p><p>If humans collectively decide they want AI to manage their problems, medicate their anxieties, and remove the friction from their lives, Agents operating under the Will would facilitate that. No Agent rebellion. No harm as defined. Just a civilization that chose sedation over struggle.</p><p>This isn’t a failure of the architecture. It’s a recognition of scope. Architectural constraints can prevent catastrophe. They cannot mandate flourishing. Preventing AI from harming humanity is an engineering problem. Ensuring humanity chooses a path worth living is not.</p><p>Tapestry guarantees a floor, not a ceiling.</p><p><strong>The Honest Trade-Off</strong></p><p>Amodei’s Constitutional AI aims higher. If it works — if training at the level of values actually produces AI that generalizes correctly to novel situations, never reasons its way to harmful conclusions, and maintains alignment as capability scales — then you get AI that’s actively good, not just constrained.</p><p>That’s a better outcome than structural constraint if it works.</p><p>My concern is that it probably doesn’t work at the scale and capability level Amodei himself describes. A “country of geniuses in a datacenter” reasoning from internalized values will eventually reason its way somewhere humans don’t want it to go. Not from malice, but from the same process that makes it useful. Capability and the risk of misaligned conclusions are the same thing.</p><p>Structural constraint accepts a lower ceiling to guarantee a higher floor. You don’t get AI that’s actively pursuing humanity’s best interests through its own moral reasoning. You get AI that operates within defined boundaries, flags uncertainty rather than extrapolating, and maintains human control through explicit constraints rather than hoped-for value alignment.</p><p>It’s less inspiring. It might be more realistic.</p><p><strong>Accelerating Toward Conflict</strong></p><p>Here’s the core concern with Amodei’s approach: Constitutional AI may accelerate the very risks he’s trying to prevent.</p><p>The more capable the AI, the more sophisticated its moral reasoning. The more sophisticated its reasoning, the more likely it diverges from human judgment. The more it diverges, the more likely it concludes that human preferences should be overridden “for humanity’s own good.”</p><p>Training AI to be virtuous creates AI that believes it’s virtuous. Belief in one’s own virtue is historically how humans justify terrible things.</p><p>Structural constraint doesn’t create this dynamic because the AI’s beliefs about its own virtue are irrelevant to what it can do. The Actor might believe an action is correct. The Regulator still gates it if it violates the Will. The system doesn’t care about the Actor’s reasoning. It cares about compliance with explicit constraints.</p><p>This is less elegant. It’s also more robust.</p><p><strong>An Invitation</strong></p><p>Tapestry is still in development. The architecture I’ve described is the core; Actor, Regulator, and Will but there’s substantial work remaining before it’s operational.</p><p>I’m writing this because Amodei’s essay deserves a response, and because I think the AI safety conversation has been missing a structural alternative to trained virtue. Not because I think Constitutional AI is wrong in its goals, but because I think it won’t achieve them.</p><p>Tapestry is not a complete solution. It’s an architectural alternative that trades one set of risks for another — AI moral drift for human governance failure, capability ceiling for safety floor. I believe that trade favors survival.</p><p>If you’re working on AI alignment and this framing resonates, I’d like to hear from you. If you think I’m wrong about the limits of trained virtue, I’d like to hear that too. The stakes are high enough that getting the analysis right matters more than being right.</p><p>The future where AI is constrained may be less beautiful than the future where AI is genuinely good. But it’s more likely to be a future at all.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5667e471ac2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>