You ask an AI agent to make a presentation. It proposes an outline, writes code to build the slides, and produces previews. One diagram is crowded. The agent splits it into two slides, enlarges the labels, and checks the result again.

That sounds like a useful office assistant. But look at what happened between the request and the finished deck: the machine helped write the method it followed.

It could also write a reviewing instruction — “check that each diagram explains one main idea” — and use that instruction in its next model call. Its output has become part of the machinery that produces its next output.

This is the intriguing turn in modern AI. Neural networks drew attention toward distributed representations and learned behavior. Generative agents bring explicit methods back into view: planning, branching, calling tools, keeping records, checking results, and revising the method itself.

Procedure returns. Who supplies the procedure begins to change.

The instruction sheet can become an object

The key connection is to Turing’s universal machine.

Imagine an interpreter that can read different instruction sheets. Give it a sheet describing one program and it carries out that program. Give it a different sheet and it performs a different computation. The interpreter stays the same; the description changes.

In Turing’s foundational work, a program can be encoded as data. That makes the instruction sheet something a machine can store, inspect, copy, and transform. A program can work on the description of another program. It can even take its own description as input.

One compact expression captures the relationship:

U(⟨P⟩, x) = P(x)

Here, U is the universal interpreter, P is a program, ⟨P⟩ is its written encoding, and x is the input. The expression says: giving the program’s description and input to the interpreter reproduces the program’s computation. If the program finishes, the interpreter returns its result. If it runs forever, faithful simulation runs forever too.

Now return to the presentation agent. A model generates a program; an interpreter executes it; the resulting previews inform another model call. The generated program may itself contain a request to the model. Descriptions and executions can therefore form a loop.

This relationship also lets the agent work on its own method. It can revise the instruction used to review slides, or propose changes to the builder program. A method becomes an object of further computation.

Language makes the loop accessible

For a language model, both a prompt and an answer are sequences of tokens. A prompt can describe a goal or a method; selected output can be placed in a later prompt to direct further work. Other output can express conventional code.

These are two useful paths. Generated instructions can guide another model call. Generated code can guide an ordinary interpreter. An agent can combine them: ask for a simpler diagram, generate code that implements the change, render the slide, and send observations back for review.

The text needs an interpretation. A natural-language request relies on the model’s instruction-following behavior, which can be imperfect. Python code has more precisely specified execution rules. The caller decides which text counts as an instruction, which is evidence, and which is sent to an interpreter.

There are established mathematical results behind the stronger universality claim. Pérez, Barceló, and Marinkovic proved Turing completeness for a specified hard-attention transformer construction. Schuurmans developed a memory-augmented language-model construction. Schuurmans, Dai, and Zanini studied universal computation through extended autoregressive decoding.

The precision, encoding, transition, and memory assumptions matter. A fixed context window by itself does not provide an unbounded working tape. Producing one token after another is insufficient to prove universality.

The practical connection is easier to see. Self-Refine reuses generated feedback for revision. CodeAct makes generated executable code an agent’s action. My argument brings these developments into a historical and conceptual account of the return to procedure; the underlying universality results belong to their authors.

The neural dream already contained procedures

The history becomes more interesting when we abandon the picture of two traditions that never touched.

In 1943, McCulloch and Pitts described neural events as capable of being “treated by means of propositional logic.” Their early neural model connected biological inspiration with formal computation.

Rosenblatt’s 1958 perceptron pursued a different emphasis. He wanted to understand recognition in systems with imperfect connections and incompletely specified organization, choosing “probability theory rather than symbolic logic.” His paper gives real historical substance to the hope that useful behavior could emerge from adaptive connections.

The attraction was clear: perhaps we could acquire behavior without writing every detail of it.

Turing had explored that possibility too. In his 1950 discussion of learning machines, he imagined an enormous direct-programming effort — sixty workers taking fifty years under his assumptions — and proposed dividing the problem into “The child-programme and the education process.” This was a thought experiment about how behavior might be acquired.

Learning was already part of the classical story. The universal machine explained how descriptions could become computations; the learning-machine discussion asked how useful behavior might be developed.

Arthur Samuel’s 1959 checkers work adds a delightful complication. His program learned to play better than its author, while combining learning with programmed search and board evaluation. He also observed that reversing a sign in its evaluation would make it improve at giveaway checkers: a game whose objective is to lose pieces.

The learner could become more successful at the objective it had been given. It would not thereby establish that this was the objective anyone wanted.

In 1986, Rumelhart, Hinton, and Williams gave an influential demonstration of learning internal representations through backpropagation. The method had earlier roots; this was a landmark in its demonstration and reception.

Backpropagation answers a credit-assignment question: when a prediction is wrong, how does the loss change with each weight? It computes derivatives backward through the network. An optimizer uses those derivatives to update parameters. Examples, objectives, and computing resources complete the learning process.

The representations are learned through an organized computational method. Neural inspiration and procedure coexist throughout this history.

The full circle changes authorship

Training on human language made requests, explanations, critiques, and revisions practically useful ways to interact with a model. For the procedural argument, the crucial fact is that the resulting behavior can help construct descriptions for further computation. Learning explains how that interface was acquired and how it might improve.

A person can specify the desired result — “make this presentation clear” — while the agent helps supply more detailed operations: split this diagram, enlarge these labels, render a preview, and revise the review instruction.

People still design controllers, define tools, set constraints, and assess results. The shift is that a complete human-written recipe need no longer be the starting point for every task.

There is also a familiar architectural parallel. Programs specify procedures over silicon circuits with substantial parallel activity. Agents specify procedures over language models whose numerical operations can be highly parallelized. The models themselves still run on silicon. Parallel implementation is compatible with procedural organization at another level.

A better deck — or a better way to make decks?

The presentation example has two possible products.

One is the deck. The other is a reusable method: render every diagram, inspect label placement, and revise crowded slides before accepting the result.

Keeping that method changes how the next presentation is made. Revising the builder program changes the agent’s operating procedure. Selected successful examples could also become training data for a later model. These are different kinds of change, even when they cooperate.

Computational reflection makes work on a method possible. Evaluation determines whether the replacement deserves to be kept. A revised procedure can be worse, and a higher score can reflect a changed evaluator.

This is why the universal-machine connection has such force: program descriptions can become inputs to programs, including programs concerned with their own operating methods.

Universal reach comes with a boundary

The ability to work on program descriptions also exposes the halting problem.

Suppose a perfect predictor could decide whether any program stops. Construct a program called Rebel. Given a description, Rebel asks whether the described program stops when given its own description. If the predictor says yes, Rebel runs forever. If it says no, Rebel stops.

Now give Rebel its own description. Either answer contradicts what Rebel does. The perfect predictor cannot exist.

This is the standard modern self-reference argument; Hamkins and Nenu discuss its relationship to Turing’s original paper. The boundary applies to unrestricted universal computation. Particular programs can still be proved to terminate, and individual runs can be bounded.

An exact universal language-model construction inherits that limitation under its specified assumptions. An ordinary chatbot getting stuck in a conversation does not, by itself, demonstrate the halting theorem.

The illuminating combination is universal execution with a limit on universal prediction. A machine can run methods that inspect and transform methods, yet it cannot provide a total, always-correct termination verdict for every possible program. There remains an indefinitely rich space of methods to construct within that boundary.

When the model changes, the resources change

So far, the model has stayed fixed while the agent constructs and revises procedures. But models themselves evolve. A slide-making controller can persist while the answering model is replaced.

This is where Turing’s oracle-machine extension becomes useful. It asks what an effective procedure can do when it has access to a specified answering resource.

For an agent, a model can play that role through a query-and-reply interface. The controller asks for an outline, a critique, or a candidate program, and uses the reply. Replacing the model changes the resource on which that procedure depends.

“Oracle” needs care here. A classical exact oracle guarantees the answers defined by its mathematical specification. A language-model service can be fallible. Calling it an oracle does not certify its answers or give it access to noncomputable information.

An effectively implemented model with a computable query allowance can be reproduced by an ordinary computation. Effective updates and revisions over the same resources can be simulated too. A newer model can make useful programs much easier to obtain without changing the class of tasks that are computable in principle.

Other researchers already use the oracle vocabulary: Laurent and Platzer’s oracular programming treats LLMs as inductive oracles within explicit computational strategies. The point here is to connect such answering interfaces with the changing dependencies of generative agents.

My relative-computability companion paper develops resource accounting and the distinction between acquiring a solver and certifying it. The distinction matters even when practical improvements are substantial.

Three questions for the next agent

The presentation task leaves us with three concrete questions. Who supplies the method? What makes the method act? What evidence justifies keeping it?

The person and model can share the first role. Controllers, interpreters, tools, and retained state supply the second. Previews, source checks, and evaluation supply the third.

Resource bounds add another practical question: can this agent find and execute an acceptable method within its allowance? Ten model calls, a minute of rendering, and a few sampled critiques define a very different claim from correctness across every admissible task. My verification companion develops these questions formally.

Turing’s framework explains the computational structure without settling every question about intelligence, competence, or reliability. Those still need empirical evidence. Its continuing power is that it accommodates neural construction, generated programs, reflective procedures, and declared answering resources in one account.

The striking development is that procedure can become something the machine helps construct. We state a goal; it proposes instructions and code; execution produces observations; further generation works on the method itself.

Procedure becomes both how the agent works and something it can create.

Adapted from my manuscript, The Return to Procedure: Language Models as Programmable Generators of Programs. The presentation scenario is an illustrative example, not an experimental result. The linked universality and agent results are credited to their original authors.

AI assistance: OpenAI’s Codex assisted with drafting and editing; the conceptual illustrations were created using OpenAI’s image-generation tool. I take responsibility for the argument and interpretation.

This concluding post brings together the four parts of the series and provides one place to access the full framework. The argument has unfolded across four layers: Part I: the outer boundary of computability, Part II: the inner boundary of feasibility, Part III: the empirical engine of scaling laws, and Part IV: the architectural response that sustains progress in practice.

The central idea is best seen directly:

At first glance, computability theory, complexity theory, scaling laws, and hardware allocation can look like separate conversations. They are not. Together they describe one layered structure of AI progress. Computability sets the outer boundary: what can be mechanized at all. Feasibility defines the inner arena: what can be reached efficiently. Scaling laws provide the empirical engine that drives expansion within that arena. Architecture then adapts to the pressure generated by that engine, sustaining the whole process in material form.

Seen this way, AI progress is neither miraculous nor contradictory. It does not mean formal limits have disappeared, nor does it mean those limits imply stagnation. It means progress unfolds through a structured interaction between boundaries, reachability, scaling, and realization. Diminishing returns have not vanished, but efficiency has become central. Hardware has not become secondary, but has become the mechanism through which scaling pressure is physically realized. That is the architecture of AI progress: a structured expansion against formal boundaries, driven by scaling, sustained by compounding efficiency, and realized by programmable architecture.

Note: This series was prepared with the assistance of AI tools, including ChatGPT, Gemini, NotebookLM, Antigravity, Cursor, and Grok, for drafting and presentation.

Why Does AI Feel Different?

We do not call it cheating to use a car instead of a horse, or a calculator instead of mental arithmetic. Yet the use of AI tools is often treated as though it were different — sometimes even as a kind of cheating. Why?

Perhaps the unease begins with how little a prompt seems to be. A few words go in; an image, an essay, a poem, or a program comes out. Too much seems to happen after too little effort.

But before deciding what prompting means, it may help to ask a deeper question: what kind of act is it to direct intelligence with language before an answer arrives?

Aristotle and the Logical Mind

To approach that question, it helps to turn to Aristotle, conventionally referred to as The Philosopher. Not because he knew anything about chatbots — he did not — but because he may have been among the first to isolate within thought something like a logical mind: a part that can be addressed, trained, and made to answer in repeatable form.

If that is right, then the story of prompting begins long before software. It begins when thought first learned to address itself in a form that could later be externalized as logic, symbolic formalism, and eventually the machine we now call the computer.

No one recorded the moment when Aristotle saw this possibility. We can only imagine it. Different arguments in his time seemed to preserve the same force even when their subjects changed. One concerned a citizen, another an animal, another a law. The content shifted, but the pattern of reasoning remained. What stays the same when the particulars are replaced?

The First Prompts

Imagine Aristotle not merely stating conclusions, but putting questions to what we might call the logical mind.

All men are mortal.

Socrates is a man.

What follows?

A citizen is subject to the law.

Callias is a citizen.

What follows?

All A are B.

C is A.

What follows?

At first the examples are concrete. But repetition strips away the particulars. Socrates matters less than the form through which he appears. The names can change; the pattern remains.

Aristotle is no longer asking only about Socrates or Callias. He is eliciting answers from an aspect of mind that responds to structure with the cold regularity of an impersonal agent.

These were not prompts in the modern sense, of course. They were not typed into a machine. But they may have been the earliest recognizable ancestors of prompting: structured inputs designed to elicit repeatable outputs from a formal part of thought.

And once thought can be addressed in that way, a deeper question arises: can it be addressed not only about the world, but about its own operations?

When Self-Reference Returns

Here the old problem of self-reference returns.

Is the statement “This statement is false” true or false?

The sentence looks simple. It is not. If it is true, then it is false. If it is false, then it is true.

What begins as a puzzle becomes a fault line in the effort to turn reason into a self-sufficient formal system. Across centuries, some of the sharpest human minds — and eventually Gödel and Turing among them — returned again and again to that fault line.

The issue was no longer only how thought could be formalized, but what happens when formal thought turns back on itself.

From Paradox to the Computer

That struggle did not produce a final resolution of self-reference. But it did help force logic into sharper and sharper explicit forms. Reason was gradually stripped down into symbolic procedures precise enough to be followed mechanically — not only by trained humans, but in principle by a machine.

Out of that transformation came something extraordinary: the computer.

And with it came another externalized aspect of mind: the programmer’s mind. What had once been implicit in acts of reasoning became expressible as procedure, instruction, code.

I tell that broader story in my previous post, From Self-Referential Paradoxes to Intelligence.

The Question We Inherit

Now, at a moment when the programmer’s mind is itself being externalized into AI, we may need to return to the very origin of computing and ask a deeper question.

If logic was invented to discipline thought, what was it in the human mind that could create logic before logic existed to explain it?

Reference

Aristotle, Prior Analytics, in The Complete Works of Aristotle, ed. Jonathan Barnes (Princeton University Press, 1984).

The First Prompts was originally published in Cantor’s Paradise on Medium, where people are continuing the conversation by highlighting and responding to this story.

Part I argued that intelligence is an open-ended ascent across computational layers. Part II argued that practical progress can proceed through staged bounded ascent within the feasible regime. Part III identified scaling plus efficiency doubling as the empirical engine. Part IV asks what that engine does to hardware design.

The central idea of Part IV is best seen directly:

Many of the most important efficiency gains — algorithmic improvements, precision formats, sparsity, software stacks, compilers, memory systems, and runtime optimizations — arrive on a cadence much faster than silicon. Even a one-year silicon cycle is already aggressive. That means a large share of effective compute growth must come not from new chips alone, but from software and system efficiency.

This is why programmability becomes central. In modern AI, it no longer means generic flexibility alone. It increasingly means the ability to absorb and optimize around model-native structures once treated as purely software concerns. Terms such as KV cache, context length, and Expert Parallelism are now part of hardware-architecture vocabulary. The table below shows how the meaning of programmability is shifting.

To see how that pressure reaches hardware, we need one more idea: the value-scalable fraction of a workload, call it S. This is not the share of work that can be parallelized as defined in classical Amdahl’s law. It is the share where adding more compute continues to create real value: better quality, new capabilities, higher accuracy, stronger reasoning, richer generation. In modern AI, that fraction is large and rising precisely because scaling laws make additional compute worthwhile across more and more of the pipeline — pre-training, post-training, inference-time reasoning, reranking, reconstruction, and other learned late-stage processes.

The first figure makes this concrete through rendering. What matters is not just that learned methods accelerate the pipeline, but that they reallocate where value comes from. Low-sample rendering becomes more like a bounded acquisition stage, while learned reconstruction absorbs a larger share of the value-producing work.

Once that shift is clear, the hardware implication follows. As the value-scalable share rises, specialization faces a rising bar. The second figure shows this directly: for a fixed efficiency advantage, the optimal hardware share devoted to specialization shrinks as more of the workload continues to reward additional compute.

The third figure generalizes the same point. It shows the threshold in (S, R) space: the more of the workload remains value-scalable, the larger the efficiency advantage a specialized unit must deliver in order to justify itself.

That changes the hardware design problem. Modern systems face a new question: how should limited hardware resources be allocated between specialized fixed-function logic and flexible programmable compute when the location of value keeps moving?

As the value-scalable fraction rises, specialization faces a rising bar. A specialized unit must be sufficiently more efficient to justify its loss of flexibility. But when value keeps emerging from evolving model structures and learned late-stage computation, that advantage becomes harder to sustain. In the limit, flexibility wins — not because specialization disappears, but because it must continuously re-justify itself against a moving frontier.

This helps explain several patterns that might otherwise seem unrelated. Why do GPUs in graphics keep becoming more programmable? Why does so much value-producing work migrate toward learned late-stage computation? Why do specialized AI accelerators matter, yet not simply eliminate the GPU? The real distinction is not GPU versus non-GPU. It is between substrates flexible enough to follow a moving value frontier and substrates over-specialized around a relatively stable mechanism.

A non-GPU design, for example, Google’s TPU, can also belong to the flexible side if it supports evolving models and workloads. The GPU, on the other hand, remains programmable, but in AI it is increasingly optimized in hardware around the demands of scaling-law-driven workloads. Across successive silicon cycles, those optimizations can compound while still preserving enough flexibility to absorb new model structures, new inference strategies, and new forms of value-producing compute. The broader lesson is simple: as scaling laws make more of the workload value-scalable, architecture is pulled toward programmable systems that can keep absorbing whatever kind of computation becomes valuable next.

A more formal treatment of this perspective, Modernizing Amdhal’s Law: How AI Scaling Laws Shape Computer Architecture, to appear on arXiv shortly.

Note: This series was prepared with the assistance of AI tools, including ChatGPT, Gemini, NotebookLM, Antigravity, Cursor, and Grok, for drafting and presentation.

Part I defined the outer boundary of what can be mechanized at all, and Part II defined the inner boundary of what can be reached efficiently. Part III turns to the empirical heart of the story: scaling laws. By now, the basic picture is familiar. As training compute increases, model loss tends to fall in a smooth, predictable way. Similar scaling behavior now appears empirically in inference, reasoning, and agentic applications as well. That is what makes AI progress seem both forecastable and relentless. But the curve also flattens. That is diminishing returns. So why has progress remained so strong?

The answer lies in a distinction that is easy to miss. In scaling laws, compute is not the same as physical burden. It is better understood as logical compute: useful work realized by the system. The real burden is time, power, memory movement, cooling, infrastructure, and operational complexity. The strength of scaling laws is that they are mostly agnostic about how that logical compute is achieved. That is why the law can stay stable even as the implementation changes underneath it.

That is where efficiency enters. Progress does not come only from spending more. It also comes from converting physical resources into logical compute more effectively. Algorithms improve. Precision drops. Sparsity is exploited. Memory systems are tuned. Hardware is redesigned. Bottlenecks are removed. The same burden yields more useful work. As diminishing returns raise the burden of progress, efficiency doubling — a broader Moore’s Law across algorithms, software, systems, and hardware — stops being a footnote and becomes the engine. The law stays the same; the figure shows how efficiency doubling beats down the calendar-time cost of satisfying it.

That is the deeper reason AI progress has remained so strong. The law stays the same, so we keep the predictability. What gets beaten down is the cost of feeding it. That is the best of both worlds: predictable progress at lower and lower cost.

And once that is true, the next question follows naturally. The pressure cannot remain abstract. It has to land somewhere concrete. It has to reshape the design of the machines themselves. That is where Part IV begins.

The central idea pf Part III is best seen directly:

A more formal treatment of this perspective is available on arXiv:

The Unreasonably Effectiveness of Scaling Laws in AI

Note: This series was prepared with the assistance of AI tools, including ChatGPT, Gemini, NotebookLM, Antigravity, Cursor, and Grok, for drafting and presentation.

From the viewpoint of computability, Part I argued that intelligence is not a fixed mechanized closure but an open-ended ascent across computational layers. But computability theory is only part of the story. Most of the action in AI is not about whether something is possible in principle with unlimited time. It is about whether something is feasible within practical resource bounds. This is where complexity theory enters. Computability asks whether something can be done at all. Complexity asks whether it can be done efficiently. That difference matters because even among tasks that are computable, only some are reachable within the bounded regime that matters in real systems. Modern AI does not live in the full universe of computable problems. It lives inside a much narrower region: the space of tasks that can be handled efficiently enough to matter.

This is why the familiar contrast between P and NP matters, but also why it is easy to misunderstand. The wrong way to think about it is: either P = NP and intelligence becomes trivial, or P ≠ NP and AI must eventually hit a wall. That framing is too crude. A better way to think about progress is as staged bounded ascent. At the most basic level, P represents tasks that can be solved directly and efficiently. NP already widens that space: it includes problems where proposed solutions can be checked efficiently, even if finding them is harder. Beyond that, the polynomial hierarchy gives a ladder of richer bounded protocols: proposal, challenge, reply, repair, and so on.

Feasibility is therefore not all-or-nothing. It has structure. AI progress does not always come from smashing through some impossible wall. Often it comes from getting better at navigating enormous search spaces through richer bounded procedures. It comes from learning how to guide search more effectively, verify more efficiently, decompose better, and avoid brute force. That is the inner boundary. The outer boundary separates the computable from the uncomputable. The inner boundary separates the computable from the efficiently reachable. Together, they give a more realistic map of intelligence: not a binary world of possible versus impossible, but a layered landscape of expanding reach. But that still leaves an obvious question unanswered: if formal structure tells us what progress means, what explains why it keeps happening?

The central idea is best seen directly:

arXiv link: to be added if and when available.

Note: This series was prepared with the assistance of AI tools, including ChatGPT, Gemini, NotebookLM, Antigravity, Cursor, and Grok, for drafting and presentation.

How formal limits, practical feasibility, scaling laws, and hardware design fit into one unified picture

You see the headlines every day. AI seems to be advancing at a relentless, unstoppable pace. But what is actually going on under the hood? We know there are hard mathematical limits to what computers can ever do. We also know that many practically important problems remain difficult even when they are computable in principle. And yet, year after year, AI keeps crossing what once looked like barriers. So what is really happening? The answer is not one magic trick. It is a layered process. At the top sits an outer boundary, defined by computability theory: what can be mechanized at all. Inside that lies an inner boundary, defined by complexity theory: what is feasible to compute efficiently. Driving progress within that feasible zone is an empirical engine: scaling laws. And the consequence of that engine is a constant pressure that reshapes the physical design of our machines: computer architecture. That, in my view, is the architecture of AI progress. It is not magic. It is a structured expansion against formal boundaries, driven by scaling, sustained by compounding efficiency, and realized by programmable hardware.

Part I. The Outer Boundary: Intelligence Is Not a Closure

To understand AI progress, we have to begin where Alan Turing began: with the question of what it means to mechanize something. In the deepest sense, mechanization means turning an activity into a finitely specifiable, rule-governed process. That is what computation is built on. But Turing’s great insight was not only to formalize computation, but also to reveal its limits. And that leads to a surprisingly important conclusion: intelligence is not a final closure. It is not a destination where we eventually automate everything and simply remain there. It is better understood as a process of climbing through layers of abstraction. Once arithmetic is mechanized, the intelligent work moves to algebra. Once algebra is mechanized, the frontier moves to calculus. Once one kind of symbolic manipulation is routinized, the human work shifts upward: choosing frameworks, defining goals, correcting procedures, and reorganizing abstractions. Every time a task becomes settled procedure, the work of guiding that procedure reappears at a higher level.

Formally, we can picture any current system as operating within a computational layer: a space of tasks that are solvable under its present rules. But every such layer has a boundary. There are always questions that cannot be answered from within that layer alone. In computability theory, one canonical way to represent that boundary is through the Turing jump. So how does progress happen? Not by one final escape from computability, but by repeated upward steps. In the computability-theorteic language of this series, such a step is a step-mechanization. Contemporary AI training lend empirical support to this picture, in which progress is not closure. It is recursive ascent.

The central idea is best seen directly:

arXiv link: to be added if and when available.

Next: Part II turns from the outer boundary of computability to the inner boundary of feasibility.

Note: This series was prepared with the assistance of AI tools, including ChatGPT, Gemini, NotebookLM, Antigravity, Cursor, and Grok, for drafting and presentation.

AI models are growing at an unprecedented pace, with breakthroughs like ChatGPT and GPT-4 transforming how we interact with technology. But as these models expand, so do the challenges of training them efficiently. Training a single large-scale model can cost hundreds of millions of dollars and consume energy equivalent to powering a small town. This raises a critical question: how do we continue advancing AI without exhausting financial and environmental resources?

Classical AI scaling laws, such as those introduced by Kaplan (2020) and Hoffmann (2022), predict training performance based on static compute budgets. These laws have guided much of the progress in AI, but they overlook key factors like time and energy efficiency. Addressing these gaps is essential if we are to sustain AI progress as models and datasets scale exponentially.

For those interested in the technical framework tackling this challenge, explore my arXiv article.

A Paradigm Shift: Efficiency Meets Time

For decades, Moore’s Law set the pace for technological innovation, predicting that the number of transistors on a chip would double roughly every two years. This exponential growth revolutionized computing, shrinking devices and boosting performance in ways once thought impossible. But Moore’s Law wasn’t just about transistors — it embodied the power of efficiency gains over time.

Classical AI scaling laws, such as those by Kaplan (2020) and Hoffmann (2022), are similar in spirit but static in nature. They predict how training performance scales with a fixed compute budget, offering a snapshot of AI progress. However, they fail to capture the reality that efficiency — whether in hardware, algorithms, or architectures — improves continuously over time.

To address this gap, I propose a dynamic, time-aware extension to these scaling laws. At its heart is the Relative-Loss Equation, which integrates time and efficiency improvements into our understanding of AI scaling.

Introducing the Relative-Loss Equation

To understand how efficiency impacts AI training over time, let’s break down the Relative-Loss Equation:

L(t) = L₀ × R(t)

Here’s what the terms mean:

• L(t): The training loss at time t , which measures how “wrong” a model is at predicting outcomes.

• L₀: The initial training loss when we start.

• R(t): The “relative loss,” which accounts for how efficiency improvements and diminishing returns shape training over time.

The relative loss, R(t), evolves as:

R(t) = (1 + (2^(γ × t) − 1) / (γ × ln(2) × 1 year))⁻ᵏ

To simplify:

• γ: The efficiency-doubling rate, such as 𝛄= 0.5 , meaning efficiency doubles every two years.

• κ: A scaling exponent that reflects diminishing returns as more compute is added.

• t : Time, measured in years.

For example, if 𝛄 = 2 (doubling efficiency every six months), training times shrink dramatically compared to slower doubling rates. For readers interested in the mathematical details, my arXiv article provides a full derivation.

This equation captures the interplay between efficiency and diminishing returns, helping us understand why efficiency gains are essential for scaling AI in realistic timeframes.

What Does This Mean?

The Relative-Loss Equation provides two key insights:

1. Small kappa (𝜅) values mean diminishing returns.

In AI scaling, kappa (κ) — the scaling exponent — represents how quickly training loss decreases as more compute is added. For large language models, 𝜅 tends to be very small (around 0.05), meaning that simply throwing more compute at the problem yields diminishing returns. It takes exponentially more resources to achieve the same level of improvement.

2. Higher gamma (𝛄) values make breakthroughs feasible.

Gamma (γ) — the efficiency-doubling rate — is the game-changer. Faster doubling rates dramatically shorten the time needed to achieve performance milestones. For example, with 𝛄= 2.0 (efficiency doubling every six months), training that might otherwise take decades can be completed in just five years.

Key Insight: The Role of Efficiency Doubling

The Relative-Loss Equation reveals a striking insight: efficiency is the linchpin for scaling AI. Without consistent gains in efficiency, achieving meaningful improvements in AI performance would be nearly impossible.

• Training time: Without efficiency gains, reducing training loss to meaningful levels would take thousands of years — even with current state-of-the-art hardware.

• Hardware demand: Alternatively, organizations would need GPU fleets so large that their energy requirements would rival those of entire continents.

By increasing the efficiency-doubling rate (gamma), these extreme scenarios can be avoided:

• Doubling efficiency every two years (gamma = 0.5) shrinks timelines to decades.

• Doubling every six months (gamma = 2.0) reduces the same milestones to just a few years.

Real-World Implications

The Relative-Loss Equation doesn’t just offer theoretical insights — it provides practical guidance for organizations, policymakers, and researchers grappling with the realities of scaling AI systems. Here’s how these insights apply to real-world scenarios:

1. Balancing GPU Investments and Innovation

Organizations face a critical decision:

• Should they front-load investments into massive GPU fleets to meet immediate performance targets?

• Or should they focus on incremental efficiency improvements to achieve sustainable progress over time?

This trade-off isn’t just about cost — it’s about long-term strategy. Front-loading hardware may seem attractive for short-term gains, but it risks becoming obsolete as efficiency improvements accelerate. On the other hand, a deliberate focus on R&D to improve efficiency can reduce costs and training timelines while ensuring scalability for future advancements.

Example:

Consider two approaches to training a large model:

• Front-Loading (the “Hare”): An organization deploys a massive GPU fleet to achieve its target quickly but at high upfront costs.

• Incremental Innovation (the “Turtle”): Another organization starts with fewer GPUs but invests in R&D to double efficiency every six months, eventually outpacing the first in cost-effectiveness and scalability.

The right balance depends on an organization’s priorities, resources, and long-term vision. My framework quantifies these trade-offs, enabling leaders to make informed decisions.

2. A Roadmap for Efficiency Doubling

Policymakers and industry leaders can use the concept of efficiency doubling to guide innovation. Just as Moore’s Law once provided clear targets for semiconductor advancements, the efficiency-doubling rate (gamma) can serve as a north star for AI scaling. Setting ambitious yet achievable goals — such as doubling efficiency every 12 months — can align efforts across hardware, software, and infrastructure development.

3. Example Goals for Efficiency Doubling:

• Hardware: Encourage the development of AI accelerators that support lower-precision arithmetic (e.g., FP8) without sacrificing accuracy.

• Software: Invest in distributed training algorithms and techniques, low-precision arithmetics, and sparsity that minimize wasted compute.

• Infrastructure: Optimize data pipelines and networking to reduce bottlenecks and improve resource utilization.

By prioritizing efficiency improvements, organizations can achieve exponential progress without running into financial or energy constraints.

Conclusion: A Race to Efficiency

This work reveals that AI scaling is not just a matter of throwing more compute at the problem — it’s a race to improve how efficiently compute is used. By embedding time and efficiency into the core of scaling laws, we gain a roadmap for sustainable AI progress.

The next decade of AI progress will depend not just on building larger models but on building them efficiently that can sustain exponential growth in a world of finite resources.

What are your thoughts on this race to efficiency? Let’s discuss in the comments!

References

1. Kaplan et al. (2020)

Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). Scaling laws for neural language models. arXiv:2001.08361.

2. Hoffmann et al. (2022)

Hoffmann, J., Borgeaud, S., Mensch, A., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. M., Welbl, J., Clark, A., Bewersdorf, J., et al. (2022). Training compute-optimal large language models. arXiv:2203.15556.

Spoiler Alert: The following review contains details that may reveal key plot points of Klara and the Sun.

This is a revision of my previous review of Klara and the Sun, updated in light of Tesla’s WeRobot event on October 10, 2024.

From Klara to Optimus
Before the rise of ChatGPT and robot assistants, Kazuo Ishiguro envisioned an AI friend who could feel deeply. In Klara and the Sun, written by the 2017 Nobel Prize winner and published in 2021, we are introduced to an embodied AI providing emotional support in a future society that feels remarkably close to our own. Given recent advances in large language models, AI companions like Klara now seem increasingly plausible, offering companionship and emotional support in ways that reflect real-world possibilities. Elon Musk’s recent announcements about Robotaxis and Optimus robots at the WeRobot event on October 10th, 2024, further highlight how AI integration into daily life is rapidly approaching, making the concept of AI companions even more believable.

A Glimpse at Klara’s World
Told from the perspective of Klara, an “Artificial Friend,” the story centers around Josie, a young girl who is ill, with her mother planning to use Klara to replicate Josie if her condition worsens. Klara, being perceptive, gradually understands the situation and decides to save Josie in her own way — by appealing to the Sun, whom she perceives as a godlike figure due to her solar-powered nature. Klara’s journey ultimately ends in the Yard, a place resembling a warehouse for unused items. Though she can no longer move freely, she retains the ability to reflect, confined to this final, quiet space.

Klara’s religious-like devotion to the Sun leads her to believe that the Sun’s intervention might help Josie recover. She also believes that destroying the Cootings Machine, an agricultural device she perceives as an obstacle to the Sun’s power, would please the Sun and contribute to saving Josie. Klara seeks reassurance from Rick, Josie’s childhood sweetheart, wanting him to confirm his love for Josie to validate her efforts. In the end, it appears that Klara has completed her mission, ultimately saving Josie.

Klara’s Perception vs. Modern AI
Klara’s perception of her surroundings as divided into ‘boxes’ offers an interesting contrast to modern AI. Unlike Klara, today’s multimodal large language models, such as GPT-4V, process both text and images holistically, understanding how different parts fit together rather than breaking them into separate segments. Klara’s approach reflects her unique, almost child-like curiosity, influencing her actions throughout the story.

Some readers may find the novel’s slow pacing and understated approach to climactic moments challenging. The reflective tone invites patience, shifting focus from action to introspection. What lingers most is the deep sense of sadness after reaching the end. Klara’s ending feels abrupt, as she transitions suddenly from her active role in Josie’s life to retreating to a utility room voluntarily and finally being confined in the Yard — a quiet and reflective space that leaves her purpose seemingly unfulfilled. Why does a story about a tool quietly fulfilling its purpose leave us with such a lingering sense of loss? Why do we feel sad about Klara’s end when we wouldn’t for an old car or a donated tool?

The lingering sadness reflects our own unfulfilled ideals — qualities like devotion, selflessness, and care are often easier to admire than to live up to. It also stems from a sense that Klara, despite her unwavering loyalty, is ultimately betrayed as her purpose becomes obsolete.

Questions Raised About Embodied AI
The novel quietly raises several questions about embodied AI:

• Should an embodied AI system be all-knowing, as modern AI aims to be, or should it retain a child-like curiosity about the world, as Klara does, which may lead to a different kind of vulnerability?
• What happens when societal inequality causes embodied AI to be available only to a privileged few?
• What potential for abuse might embodied AI face? Klara is subjected to physical abuse, such as when Josie’s friends bully her and threaten to toss her around, much like the verbal abuse directed at today’s chatbots. Imagine what could happen when Tesla’s Optimus robots are around us.
• How do we manage the hallucination-like decisions that arise from limited knowledge, as seen in both Klara and modern AI systems? These misguided actions carry potential dangers that we may not yet fully understand.
• How should we handle the beliefs of an embodied AI? When an AI reaches the highest level of human reasoning — counterfactual thinking, as Judea Pearl described in his work on causal inference (Pearl, 2018) — it may start to think about things that do not exist. Klara’s belief in a god-like Sun is an example of this, raising questions about how such beliefs should be managed or interpreted.
• How should embodied AI interpret and make decisions from what it learns? Klara observes and distills knowledge from being among humans, reflecting on their actions and emotions. This raises questions about how embodied AI might interpret observations and how it might make decisions based on the insights gained from its experiences.

Conclusion
The motivations behind characters like Klara might differ from those driving real-world AI development. With all the enthusiasm for AI’s problem-solving prowess, one wonders: Will embodied AI be super-smart, as Sam Altman envisions, or witty, like Elon Musk? How can traits like empathy, selflessness, or devotion — the qualities Klara embodies — find a place in our current technological landscape?

References
Pearl, J., & Mackenzie, D. (2018). The Book of Why: The New Science of Cause and Effect. Basic Books.

Spoiler Alert: The following review contains details that may reveal key plot points of Klara and the Sun.

You can see the revised version here.

Introduction

Kazuo Ishiguro’s *Klara and the Sun*, written by the 2017 Nobel Prize winner in Literature and published in 2021 — before the frenzy around ChatGPT and large language models — depicts an embodied AI providing emotional support in a future society.

*Klara and the Sun* portrays an AI companion in a future that feels remarkably close to our own. Given recent advances in large language models, AI companions like Klara now seem increasingly plausible, offering companionship and emotional support in ways that are beginning to reflect real-world possibilities. Elon Musk’s recent announcements about Robotaxis and Optimus robots at the WeRobot event on October 10th, 2024, in Los Angeles further highlight how AI integration into daily life is rapidly approaching, making the concept of AI companions even more believable.

Plot Summary

Told from the perspective of Klara, an “Artificial Friend,” the story centers around Josie, a young girl who is ill, with her mother planning to use Klara to replicate Josie if her condition worsens. Klara, being perceptive, gradually understands the situation and decides to save Josie in her own way — by appealing to the Sun, whom she perceives as a godlike figure due to her solar-powered nature. Klara’s journey ultimately ends in the Yard, a place resembling a warehouse for unused items. Though she can no longer move freely, she retains the ability to reflect, confined to this final, quiet space.

Klara’s religious-like devotion to the Sun leads her to believe that the Sun’s intervention might help Josie recover. She also believes that destroying the Cootings Machine, an agricultural device she perceives as an obstacle to the Sun’s power, would please the Sun and contribute to saving Josie. Klara seeks reassurance from Rick, Josie’s childhood sweetheart, wanting him to assure her that her efforts are worthwhile by confirming his love for Josie. In the end, it appears that Klara has completed her mission, ultimately saving Josie.

Klara’s Perception vs. Modern AI

Klara’s perception of her surroundings as divided into ‘boxes’ offers an interesting contrast to modern AI. Unlike Klara, today’s multimodal large language models process both text and images holistically, understanding how different parts fit together rather than breaking them into separate segments. Klara’s approach reflects her unique, almost child-like curiosity, influencing her actions throughout the story.

Some readers may find the novel’s slow pacing and understated approach to climactic moments challenging. The reflective tone invites patience, shifting focus from action to introspection. What lingers most is the deep sense of sadness after reaching the end. Klara’s ending feels abrupt, as she transitions suddenly from her active role in Josie’s life to being confined in the Yard — a quiet and reflective space that leaves her purpose seemingly unfulfilled. Why does a story about a tool quietly fulfilling its purpose leave us with such a lingering sense of loss? Why do we feel sad about Klara’s end when we wouldn’t for an old car or a donated tool?

The lingering sadness reflects our own unfulfilled ideals — qualities like devotion, selflessness, and care are often easier to admire than to live up to. It also stems from a sense that Klara, despite her unwavering loyalty, is ultimately betrayed as her purpose becomes obsolete.

Questions Raised About Embodied AI

The novel quietly raises several questions about embodied AI:

1. Should an embodied AI system be all-knowing, as modern AI aims to be, or should it retain a child-like curiosity about the world, as Klara does, which may lead to a different kind of vulnerability?
2. What potential for abuse might embodied AI face? Klara is subjected to physical abuse, such as when Josie’s friends bully her and threaten to toss her around, much like the verbal abuse directed at today’s chatbots. Imagine what could happen when Tesla’s Optimus robots are around us?
3. How do we manage the hallucination-like decisions that arise from limited knowledge, as seen in both Klara and modern AI systems?These misguided actions carry potential dangers that we may not yet fully understand.
4. How should we handle the beliefs of an embodied AI? When an AI reaches the highest level of human reasoning — counterfactual thinking, as Judea Pearl described in his work on causal inference (Pearl, 2018) — it may start to think about things that do not exist. Klara’s belief in a god-like Sun is an example of this, raising questions about how such beliefs should be managed or interpreted.

Conclusion

The motivations behind characters like Klara might differ from those driving real-world AI development. With all the enthusiasm for AI’s problem-solving prowess, one wonders: will embodied AI be super-smart, as Sam Altman envisions, or witty, like Elon Musk? How can traits like empathy, selflessness, or devotion — the qualities Klara embodies — find a place in our current technological landscape?

References

Pearl, J., & Mackenzie, D. (2018). *The Book of Why: The New Science of Cause and Effect*. Basic Books.