What to Expect When You Are Expecting Recursive Self Improvement

There is a new buzzword that is taking the world of AI by storm: Recursive Self Improvement (RSI). See for example, this blog post from Anthropic as a representative example. As with other broad terms in AI such as Artificial General Intelligence (AGI), Artificial Super Intelligence (ASI), and even the term Artificial Intelligence itself, RSI doesn’t have one definition that everyone agrees upon. RSI captures the general notion of an AI system that creates a version of itself that is improved, which in turn creates a version of itself that is further improved, and so on.

The interest in the term RSI has stems from reports by so-called “frontier” AI companies such as Anthropic and OpenAI that 80% or more of the codebases of the Claude AI system and the GPT AI system were written by Claude or GPT, respectively. In the last year, AI systems have gotten very good at writing code. It only makes sense that a software company that makes AI models would use AI to write their code. Naturally, it becomes inevitable to ask: if an AI system is writing the code for the next version of the AI system, and that new version is better, then what happens when the AI system is capable of writing 100% of a better version of itself?

The term, recursive self-improvement, may be new, but the concept is one that has been around in various other guises. In 1965, mathematician I.G. Good speculated on an “intelligence explosion” that would arise from “ultraintelligence” machines:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion’, and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.

This is also referred to as the Singularity, which captures the hypothetical notion of technological growth accelerates exponentially and eventually reaches a point where growth is immeasurably fast and thus beyond human ability to understand it or control it.

A singularity occurs when the rate of change in intelligence becomes infinite.

The singularity comes in many flavors. Technology advancement can take the form of a hard take-off or a soft take-off. More on this later. Depending on one’s priors, this can be a very exciting prospect that heralds the “expansion of intelligence throughout the lightcone of the universe” or the AI apocalypse.

Before we dig into the hypotheticals of the singularity, it is important to ground ourselves in what RSI looks like now in order to get a sense for how we might get to a hard take-off or soft take-off scenario (or no take-off). To do this, we must familiarize ourselves with what it means to build an AI system in the first place.

What is AI (to a Builder)?

This is not going to be a primer on AI or machine learning algorithms, neural networks, or large language models. If you want a background on LLMs, you can see my Very Gentle Introduction to Large Language Models without the Hype, or the even gentler Intuition Behind How Large Language Models Work.

Instead, we will pull back the curtain on what it looks like to build an AI system. It is easy to slip into a mindset of thinking about AI as a monolithic thing, that lives on a computer or in the cloud, like a brain living in a body. We only see what it does with the inputs we give it, so we are naturally keyed into its behavior.

However, artificial intelligence are built things. And as with all built things, there are hundreds or thousands of small and large decisions that are made along the way.

There are many different types of AI algorithms. We will focus on Machine Learning algorithms, and in particular the neural network architectures behind Large Language Models because learning systems are driving a lot of progress in AI right now. Machine learning systems, such as Large Language Models (LLMs), derive their capabilities from data. A simple description of machine learning is any system that extracts patterns from data and then derives behavior from those patterns. In the case of LLMs, the raw data is, for the most part, raw text, and the behavior is generated text in response to a prompt.

Those who build LLMs have taken to referring to their method of acquiring data, preparing data, and training the model as a “training recipe”. We can say we are baking a cake, but that isn’t precise enough to be actionable. What kind of cake? How big is the cake? What flavor? What ingredients? How much of each ingredient? How long in the oven? What will be used to decorate? What will those decorations look like? To say we are building an LLM, we must also know:

• What is the high-level architecture (typically the Transformer neural architecture)?
• What are the low-level architectural details such as the number of layers, the size of each layer, how attention is implemented, etc.?
• What data will be used and how have those data been cleaned and preprocessed in advance of training?
• How long will the model be trained, and will there be different phases of training involving different mixes of data?
• Models don’t run themselves; how will the execution harness operate once training is complete?

The way each of these questions (and more!) are answered will have profound effects on the quality of the model and the quality of the user experience using the model. To build a model is to make decisions about every stage in the recipe. Some of these decisions are based on trial and error over decades (if not more) of research. Some of these decisions are made arbitrarily because they seem to work well enough for now.

The Ingredients

In this section we are going to look at some of these decisions that must be made when building an LLM. We will look at high-level and low-level architectural decisions, as well as data decisions, the training harness, and the execution harness.

High-Level Model Architecture

There are many ways to build a language model: probabilistic graphical models, recurrent neural networks, long short-term memory networks, Transformers, diffusion, etc. Each technique has pros and cons with respect to their ability to extract patterns from data. For example, a probabilistic graphical model is going to work really well on small, specialized vocabularies and require much less computation and training data than a neural network. Neural networks excel at larger language spaces but require extraordinary amounts of data and computation for training and execution. Recurrent neural networks can have theoretically infinite context windows (though in practice that hasn’t been the case), whereas transformers have fixed context windows.

Right now, almost all production-grade LLMs are based on the Transformer neural architecture. Architecture here just means the general configuration of computations that are conducted inside a neural network when it processes data. In this case, what makes a neural network a transformer is a particular configuration of layers with learnable parameters and layers that perform attention computations. Sort of like how one gets different types of layer cakes based on the type of dough and whether there is creme, fondant, or custard mixed into the layers.

You can take a look at this high-level description of how Transformers work for more information.

Transformers aren’t the only option. Diffusion models are being developed for text and have been shown to have some promise, as well as state-space models which are extensions of recurrent neural networks. However, we will assume the transformer architecture going forward.

Low-Level Model Architectural Details

Once a high-level architecture is chosen, there are a lot of details that must be chosen. I enumerate some of those that must be chosen for transformer models below.

(If the list below looks daunting, feel free to skip over it. The point is to show that there are a lot of details that LLM engineers nerd out over.)

• Layers. How many layers (how “tall” the model is).
• Layer width. How many parameters per layer (how “wide” the model is).
• Mixture of experts. Most very large AI systems are made up of many smaller transformers, each trained on slightly different mixtures of data to specialize on. How many experts will be used, and what data mixture is used for each must be chosen. A router is another sub-systems that chooses which transformer will respond to your prompt. How the router is implemented — whether hard-coded or trained — is another choice.
• Tokenizer. Neural networks operate on numerical data, so text must be converted into numbers (called tokens). This process is called tokenization. Common words are converted to single numbers, and uncommon words are split up into multiple tokens. How this tokenization happens has been shown to have an effect on how LLMs understand our prompts (e.g., shopify → shop+ify, vs. shopify → sho+pify). Tokenization is one reason why LLMs don’t know how many ‘r’s are in the word strawberry; they don’t see letters, they see numbers.
• Context window. The context window is a buffer that contains the prompt and any additional tokens that are generated after the prompt is provided. The size of the context window determines how long a prompt can be, and how many tokens can be tracked before the LLM starts to forget the prompt and any work that it has done.
• Attention Computation. The attention computations in an LLM are really expensive — the cost is the size of context window raised to the power of two. A technique called FastAttention reduces the cost by caching redundant computations.
• Positional encodings. LLMs need to keep track of what order the tokens are in, so small perturbations are added to the data depending on where they are in the context. The exact set of mathematical computations used to do this can be done in many different ways, making LLMs more or less sensitive to word order.
• Normalization. Neural networks operate on numerical data, those numbers can get too big or too small. Normalization layers shift the numbers to keep them in a typical range. The original transformer architecture used something called layer normalization and newer implementations use root mean square normalization, which has been shown to work better. The details aren’t important; there are many ways to do normalization.

There are likely more that I am not aware of, since I do not, as part of my research, investigate these types of details in LLMs.

Data

Machine learning systems learn behavior/responses from patterns in data. However, we don’t just grab data off the internet and chuck it into a learning algorithm. There are different data that are used at different stages of training. Pre-training uses text data from books, the internet, and other sources. Mid-training uses specialized text prompt-response pairs to make the LLM better at responding to requests. Post-training uses numerical data like prompt-response ratings to reward or punish LLM responses (called reinforcement learning).

• Pre-training data curation. What data gets collected, and from where? (Is it compliant with copyright laws?) Some data can be removed because of its sensitive nature. Data must be de-duplicated because some documents and text spans can occur multiple times in the data.
• Mid- and post-training data curation. Mid- and post-training data can include human feedback data where people manually correct LLM responses and provide quality ratings. What examples should people see? Which ones should they provide feedback on and which ones should they rate? How many people need to see each example? What are the instructions that people receive when providing feedback or ratings? How much should people be paid for piece of data (assuming data quality is correlated with pay)? When correcting code, should people also provide justification and documentation of their edits? These methodological decisions that go beyond what to scrape and how to clean data can have big impacts because they shape the behavior of LLM responses.
• Data mixture. Each expert in a mixture of experts set-up needs different mixes of data. A more mathematically oriented expert probably doesn’t need poetry, but maybe it needs a basic set of natural language in different languages to acquire fluency to understand a math word problem.

Training Harness

Now that you have an architecture and all the implementation details pinned down, and you have your data, you must implement a training loop.

• Epochs. How many times through the data?
• Batch size. GPUs can work on data in parallel in clusters called batches, but error (called loss) is computed across all data in a batch, meaning that loss of a single data point is an average of loss of all datapoints currently being processed.
• Batching strategy. Which data are batched together?
• Data quality schedule. LLMs benefit from switching between training on high-quality data and low-quality data. What constitutes high- and low-quality data? How should high- and low-quality be mixed together? How often should the training loop switch from mixtures that are primarily high-quality to mixtures that are primarily low-quality?
• Learning rate decay. How much the parameters of a model are allowed to changes after being exposed to each data point — called the learning rate — can help the LLM find good values for parameters. Too high and the LLM will change too fast and might skip over good values. Too low and the training will take longer and require more passes over the data. Generally, one wants the learning rate to be high early and low later on. How fast should the learning rate be adjusted?
• Sampling. For post-training that uses reinforcement learning, the order in which data samples get ranked matters (PPO, GRPO, etc.; what these acronyms mean is not important).
• Verifiable rewards. One of the reasons LLMs have gotten so good at coding, math, and logic is because coding, math, and logic are verifiable — one can look at the problem and determine if the answer is correct. There are special programs that have been built that do verification in these domains. They can be used to automatically reward the LLM during training on these problems. The choice of verifiers and how they are used in training can have big impacts on model quality. However, new verifiers must be built if they don’t already exist.

Execution Harness

So you have a model. Model’s don’t do anything by themselves. And they only generate one token at a time. The most basic execution harness runs a loop wherein the model is fed a prompt, the model produces a set of scores over all possible next tokens, a token is selected, that token is added to the prompt (now called a context), and the process continues.

Get Mark Riedl’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Here are some of the things that go into execution harnesses:

• Prompt filtering. The prompt is checked for appropriateness and possibly rejected before ever being fed into the model.
• Prompt modification. The user’s prompt is combined with a system prompt — a secret set of instructions that get prepended or appended to the user’s prompt to shape the model’s response.
• Skill files. A skill file is a system prompt for solving a specific type of problem. The user’s prompt is inspected to see if they are trying to do a specific tasks and if there are any skill files associated with those tasks.
• Response filtering. The output from the model may be checked for appropriateness and suppressed.
• Tools. Tools are external program that the LLM can call to perform specific tasks (search the web, check the weather, search a file for a keyword, etc.). The LLM may be trained to generate a special token strings to invoke a tool (e.g., “{tool: weather, location: …}”). The execution harness detects those special token strings and initiates the call to the program and substitutes the results back into the LLM response. What? You thought this happened inside the neural network?What tools are made available to the LLM and how those tools, which themselves are code, are implemented can make AI look really smart.

What we think of as “the AI” is really a system comprised of a model and a whole bunch of stuff that is not the model but regular, standard code.

Cooking with Hardware

Once one has decided on all the ingredients, one must start baking. Training neural networks is very computationally expensive: the transformer architecture choices that work best involve very large models (current models are likely in excess of 10T parameters), and datasets are in the order of documents comprising trillions or tens of trillions of tokens. The GPU is the powerhouse that makes it feasible to train models of this size. One wants as much RAM local on each GPU as possible, and one wants really fast throughput. While we normally think of throughput as a function of the hardware, the software that moves data in and out of the GPU can also be optimized. Recently, Claude Fable 5 was used to rewrite the software that moves code on and off the GPUs to achieve up to an 18.7x performance improvement in some cases. This doesn’t make the model “smarter”, but it matters a lot to a company that values response times during execution and training time of new models.

Taking Stock

Some of the ingredient and hardware optimizations listed above are good for a company’s bottom line because a model with the same — or similar — capacity runs faster and/or uses fewer computational resources.

Some of these decisions make the AI system seems smarter without fundamentally changing anything about the learning system. For example, better retrieval will reduce errors and better skill files in the execution harness will tell the system how to compose better responses to specific, tasks. These improvements can be achieved without changing a single parameter in the model itself.

Some decisions will make the core model “smarter”, including:

• Better data
• Better utilization of data
• Better reinforcement learning sampling algorithms and better rewards
• Some low-level architectural details such as the right number of layers, the right size of layers, and the right number of MOEs.
• High-level architectural details including radically new architectures that replace the transformer

Right now, the transformer, however, is the best option we know about. There are many implementation details that improve on the basic design, like MOE. However, the basic design has been fundamentally unchanged since 2017.

The current state of affairs in the summer of 2026 is that RSI readily capable of addressing design choices involving low-level architectural details, hardware optimization, and some aspects of the training and execution harnesses, and some aspects of data utilization. There is potential for an AI system to design new and better data collection methodologies as well (AI optimization of data collection pipelines has been an active area for more than two decades).

What gets people excited (and possibly concerned) about RSI is the prospect of an AI system that makes an AI system that is “smarter” and more capable. The question will inevitably arise as to whether an AI system can discover and exploit a new high-level architecture that outperforms the transformer.

It is not impossible that a new architecture exists that can supplant the transformer. One possibility already known is something called the state-space model, which has some interesting properties that we would like to have such as cheaper execution costs and theoretically infinite context windows. (State-space models are actually more similar to older types of language model called Recurrent Neural Nets and we just didn’t know how to make them big enough). So far state-space models haven’t quite been shown to scale to the same performance as transformers on the same data, but maybe another 10 years or work might see that happen, in which case existing AI systems should be able to help us speed-run that process.

It is also not impossible that some superior high-level architecture exists that humans have not yet thought of. By “superior” I mean an architecture that can pull more/better patterns out of the same data compared to the transformer? The space of permutations just within neural networks is near infinite. Most will be nonsense or inferior to the transformer because if the space of all possible neural architectures had a lot of better options, we probably would have found it already or have some theoretical inkling of its existence; it will be a needle-in-the-haystack search. The thing to remember is that the transformer didn’t come out of no-where, it was the evolution of 10+ years of human insights building off each other.

Could Claude or another AI system theoretically crunch through trillions of possible new architectures, and train them to the size of modern transformers on full multiple-trillion data-point data? Yes. Would it be costly beyond reason? Also yes. The entire world does not currently have the computational infrastructure capability to do this at anything other that toy levels. This is certainly true while most infrastructure capability is going to inference, serving users of current models.

We will look at the some of the work on AI systems designing AI systems in a later section.

One of the reasons that RSI can evoke concerns is the prospect that an AI system not only comes up with a new type of AI system that is superior, but it also comes up with something that humans cannot understand. Some scientists like to pithily state that we already do not “understand” how our current models work. What is really meant by that is that we do not know precisely how the particular set of parameters that transformed the prompt into a response were set the way they were during training. We can look at those parameters — they are just floating point numbers — and we can trace the mathematical computations that produced a response. At the architectural high level we “understand” transformers quite well.

Could an AI system develop a new type of model that we didn’t understand? That is the question. An AI system might come up with something that surprises us. That is almost a given because we haven’t already thought of something better. But we would be able to understand it because we can always understand code; a model’s architecture must first exist as code that specifies the existence, shapes, and sizes of learnable parameters, as well as the training harness code that accompanies the model. Also remember: Claude and other current models learned how to write AI code from existing AI code written by humans. It will likely start with the same intuitions that we have expressed through our writing and documentation that ended up in training data. Any search through the unending set of permutations will be more efficient if it starts with things that are known to work and it will likely use intuitions it learned from us to make educated guesses about what might also work and be better.

What the AI system could potentially do is to also make more radical explorations that a human might not choose in manual experimentation. After all, human experimentation would be slower and humans have a finite lifespan (and career span) so a particular conservativeness is valued if one expects to see results in one’s lifetime. But AI isn’t free either, so a balance of likely trials versus radical alternatives will be necessary even for AI systems running autonomously. Thus, we should not expect an AI to find what humans couldn’t. What we should expect is possibly that discoveries happen faster because AI can work in parallel and doesn’t need to sleep or eat.

We Should Expect Diminishing Returns

Whether talking about lower-level architectural, data, and hardware optimizations, or about finding new architectures, we should expect diminish returns over time.

Low-level implementation details will eventually converge on the most efficient. For example, there are only so many ways to implement attention computations. Computation theory tells so that some processes cannot be completed under certain minimum amounts of time (a process that naively takes n² computations where n is the number of tokens in the context window might be made to take only n computations through efficient hardware caching, but can never be sub-linear). Hardware optimizations have a floor where physics cannot push more electrons through circuits.

Skill implementations in the execution harness may provide a period of perceived capability increase but eventually skills will get harder to improve. New skills can be made for more tasks, but after the most common tasks have good skills, new skills will only be able to address increasingly niche tasks.

We might make better choices about MOE configurations, number of layers, size of layers, etc. However, the bottleneck will be the data that we have. Getting new data is expensive and hard and only improves in more specialized use cases. New data is hard to get. Data can be created by people at the speed that people make data. People generate a lot of words on the internet for free. But it’s not necessarily about what an LLM would need to get better at things, which is why most AI companies are paying for specialized data to be created. This is slower and more costly. Data can be collected from the physical world via sensors and robots, but these are also constrained by how many sensors, how many robots, and how fast the robots can accomplish things. I’ve described the data bottlenecks elsewhere.

What about verifiers and simulators? New verifiers can help steer an AI learning algorithm toward higher quality patterns. However, verifiers have to be created and they must be validated against the real world, which runs into the same bottlenecks as before. Simulators can generate synthetic data and, if done well, generate data faster than it can be collected from the real world. However, the simulators must be created and once again validated.

In short, we’re tapping out on cheap data and new data is more specialized and harder to get. We are not tapped out on specialized data, but it comes at a slower pace. The data that needs to be collected will need to become more and more bespoke, addressing more and more niche tasks, and thus be harder and more costly to acquire because it will require specialized skills by humans or robots to produce it.

One way out of the diminish returns spiral is new high-level architectures that can somehow extract higher quality patterns and understanding from the same data that we are currently using. That is presupposing that the data we have is much more laden with patterns that current transformers are not finding and exploiting. This is not entirely unreasonable; the human brain seems to be pretty good at pulling patterns from data (at least the patterns that it needs to function in the world). It’s dangerous to make comparisons between computational AI models and biological brains (for a lot of reasons I will not go into here. Besides, we know how to make brains. (We are very good at making brains. Most everyone can do that for relatively cheap — it costs approximately $300,000 spread over 20 years to raise kid to maturity and educate them.)

Hard Take-Off?

Also called foom (for the sound it of something accelerating toward infinity. A lot of venture capital is driven by foom fomo.)

Between diminishing returns of low-level recursive improvements, data bottlenecks, and the costs associated with of radical experimentation with new neural network architectures, it is my belief that hard take-off is hard to justify.

Soft Take-Off?

What about a soft take-off? A soft take-off is one in which AI system capability recursively improves, but at a soft pace where humans are able to keep up with, and understand, each improvement. I would argue that nothing AI will come up will be something that we could not understand (the brain, for example, is one of those things that we do not understand particularly well). In a soft-take off scenario, humans will be capable of watching and learning from computers and in turn be smarter about how to make artificial intelligences; if something does surprise us, that surprise should be relatively incremental and we will have the time to acquire understanding.

Hard vs. slow take-off.

Under the soft take-off scenario, it is likely we can then design our own intelligences that are more under our control. After all, what can an AI do that humans will never understand? Even if an AI system can store more in its “mind” simultaneously than any single human ever could, we have the means to offload computation through computational crutches like writing, or make tools. Arguably, this is potentially where we are right now.

But we humans are slow. Perhaps even slow RSI results in some Rubicon being crossed where human understanding falls behind in a way that we cannot catch up. This is the nature of exponential curves; at some point the elbow is turned. However, the notion of slow take-off includes the possibility of humans intentionally halting the recursive self-improvement process at any point. For an extended period of time we will have the ability to intervene in the RSI process, even after said Rubicon is surpassed. Interventions can include via regulation of the use of recursive improvement, shutting down of servers, turning off power to servers, disabling internet infrastructure, EMP, nuking data centers, all of which have been proposed at one time or another.

However, long before we get to that, we will see things coming. We will be able to observe and measure power usage, autonomous power-plant creation, data center usage, autonomous data center creation, and autonomous creation of new hardware fabrication facilities. None of these things are instantaneous and have footprints in the physical world that would be hard to hide.

No Take-Off?

In nature, there are very few true mathematical singularities. Most things are finitely-bounded by energy, resources, or other limiting phenomenon. A lot of things in nature look exponential but are s-shaped curves in the long-run when a theoretical maximum is approached. We might call this the no take-off scenario, in which there is an exponential phase that levels off. Diminishing returns take effect. (The term no take-off is a bit of a misnomer because it is more of flying-at-max-altitude). In this scenario it is less critical that AI capability temporarily exceeds human ability to keep up in the steepest part of the curve because humans will have time to catch up later.

If there are natural limits to AI capability then progress must slow as it asymptotically approaches the limiting factor.

AI Designing AI

This is totally a real thing that has been around for a while. The most common form is in some sort of AI system that designs a completely different type of AI system. What is potentially new is the AI designing a version of itself — this wasn’t really practical until LLMs could code. Since AI is code, then an AI that can code can improve the code that makes up an LLM.

Here are a couple of ways that AI has designed AI in the past:

• Evolutionary algorithms. In evolutionary computing, an algorithm starts with a random set of configurations of something — called individual instances — and measures their fitness — how well each does on some performance metric. Changes are made to the individual instances, either by randomly changing some aspect of each or by trying to identify what made them good and copying it into other instances. The fittest are kept and the failed experiments are discarded. You can do this with neural networks. It has been proven to find variations of an original model that is more specialized and more performant on specific tasks. Usually this variation is found to have a slightly different size of layers or depth or have different connectivity between layers. The evolutionary computing process can be considered an AI system, though not necessarily one that learns. It is AI in the same way that car navigation is AI: an algorithm that optimizes a seemingly intractable problem in a reasonable average time.
• Self-play reinforcement learning. Self-play is the reason why AlphaGo was able to beat human expert Go players. Start with some agents based on neural networks that all use the same architecture but are initialized slightly differently so they learn slightly different things when they play a game. Have the agents play against each other. Take the best agent and “freeze” it so it cannot learn any more. Train the other agents to beat the current best through trial and error. Rinse and repeat. Unlike evolutionary computing, self-play does not evolve new architectural designs, but instead experiments with data acquisition in the game playing setting where a simulation exists and an agent can engage with the simulation in different ways in order to learn what it needs to win (data in this case is how the simulation responds to different actions in different states). Like evolutionary computing, the self-play meta-loop can be considered an AI algorithm that searches the space of trained agents for the best one.

One thing to note about both evolutionary computing and self-play is that the harder the problem is, the more trials are needed. Each individual instance must train, which is a costly process. Each loop of evolutionary computing or self-play is thus costly and time-consuming; training anything other than a toy-sized can take hours, days, weeks, or months. And that is just one loop; evolutionary computing and self-play loops can require hundreds or thousands even for small problems.

Evolutionary computing and self-play are not recursive in the sense that the algorithms themselves are not improving themselves, but are improving something else that is different from then. It is a simpler AI creating something both different and more sophisticated (in a different way) than it is. However, the principle applies to RSI where a coding LLM is writing code that trains and runs a coding LLM. The LLM is implementing the outer loop (or writing code for the outer loop) and a new version of itself is the product of each iteration.

One thing that a LLM implementing RSI loop can do that evolutionary computing and self-play cannot do is to adapt its own way of implementing the RSI loop. For example, a sufficiently sophisticated LLM could look at the progress being made and propose different pseudo-random permutations based on what is most likely to have impact from that point, or make smarter choice about when to exploit some intuition versus try something random. Of course, those intuitions are learned from us so it is hard to know, up to a certain point, if it would be making the same iterative changes that human scientists would make. It may also think up new tests as it iterates to refine the notion of fitness based on emergent capabilities. But it is unclear whether such optimizations to the RSI loop will provide significant advantage; the evolutionary computing loop and self-play loops are relatively simple because iterating fast is often better for making forward progress than slower and “smarter” in the long run (however, there is a trade-off; when each iteration is costly then it is sometimes better to have a slower iterations that are more likely to make forward progress). Plus experimenting with the RSI loop also means experimenting with different RSI loop configurations, some of which could be worse.

Conclusions

The fact that LLMs can code LLMs is exciting. There will be a near-term acceleration of progress. It is already happening. However, at the time of writing RSI seems to be more about finding incremental optimizations that improve computational efficiency. This is a good thing. We want highly capable models that require less compute so that we can offer their services cheaper, faster, and more widely. The things that are going to make LLMs significantly “smarter” in the long-term are going to be new data and new high-level architectures. These are not likely to come easily even with RSI.

RSI naturally leads one to wonder about the singularity and hard take-off in particular. A hard take-off is easily conflated with existential risks to humanity (although even a hard take-off doesn’t guarantee that a super-powerful AI will seek human extinction) and it is easy to see how questions of RSI can slip into questions of hard take-off and human extinction.

However, based on the analysis in this article, if any notion of the singularity is to occur, then a slow take-off is more likely. Slow take-off is a more controllable scenario with a lot of points for human intervention.

We may be in the middle of a “no take-off” scenario, in which AI capabilities asymptote before they reach the level where humans can no longer understand and keep up. In some ways, it doesn’t matter too much if we are in a soft take-off or an asymptotic scenario as the questions of safety, control, and governance will be much the same.