Understanding Model Collapse
Why model collapse matters, what it looks like in practice, and how we can keep the AI ecosystem from becoming a hall of mirrors
There is a strange paradox at the heart of generative AI.
The more successful these systems become, the more content they produce. That content spreads across the internet: articles, product descriptions, illustrations, code, summaries, reviews, tutorials, social posts and even scientific-looking explanations.
Then someone comes along and trains the next generation of models on the internet.
That sounds reasonable. It is, after all, how much of modern AI has been built: collect enormous quantities of information, filter it, and train increasingly capable models.
But there is a problem.
What happens when a significant portion of that information was produced by the previous generation of AI?
The answer may be model collapse.
And despite the dramatic name, model collapse isn’t a science-fiction scenario in which a chatbot suddenly stops working. It is subtler — and potentially more important.
A model can gradually lose contact with the richness of the original world it was trained to represent.
The basic idea
Imagine that you are trying to learn what the world looks like by reading everything you can find online.
At first, your sources are mostly human.
You encounter unusual opinions, obscure historical events, regional expressions, badly written restaurant reviews, niche hobbies, strange personal experiences and countless other details that make human culture messy and diverse.
Now imagine that another person summarizes all of those sources.
You read their summaries instead.
Then a third person summarizes those summaries.
Then a fourth person summarizes the third person’s summaries.
The information may remain broadly correct for a while.
But something starts disappearing.
The unusual details disappear first.
The edge cases become less common.
Ambiguous information gets simplified.
Mistakes can be repeated.
Eventually, the summaries begin describing a world that is cleaner, narrower and more predictable than the original.
This is roughly the intuition behind model collapse.
In a 2024 study published in Nature, researchers showed that repeatedly training generative models on data produced by earlier generations can cause the models to progressively lose information about the original data distribution. The researchers observed the phenomenon across language models as well as other generative-model families.
The important word here is repeatedly.
Generating synthetic data isn’t inherently bad.
The danger appears when synthetic data becomes a substitute for the underlying human-generated data and is then fed back into the training process again and again.
That creates a feedback loop.
Human data → Model A → Synthetic data → Model B → Synthetic data → Model C → …
Eventually, the model is no longer learning primarily from the world.
It is learning from its predecessors’ interpretation of the world.
The disappearing tail
One of the most useful ways to understand model collapse is through something statisticians call the tail of a distribution.
Suppose you ask people what they like to eat.
Pizza might be extremely common.
Pasta is common too.
Rice, burgers and sandwiches are common.
But then you get a long list of less frequent answers: a regional dish from someone’s hometown, a family recipe, something they ate while travelling, an unusual combination of ingredients.
Those rare answers are part of reality too.
A model trained on enough human data can potentially learn about them.
But a generative model tends to produce things according to probabilities. Common patterns are easier to reproduce than rare ones.
Now imagine generating a huge dataset with that model and using the generated dataset to train another model.
The rare examples have a problem.
They were already underrepresented in the first model’s output.
The second model sees even fewer of them.
The third sees fewer still.
This is one of the mechanisms behind model collapse: the tails of the original distribution can disappear across generations. The Nature study found that models increasingly concentrated on high-probability patterns while also accumulating errors that were not present in the original distribution.
In other words, the model can become more statistically confident while becoming less faithful to reality.
That is a dangerous combination.
A simple example: the AI travel writer
Consider something completely ordinary.
A travel website publishes thousands of articles about Paris.
Some are written by professional journalists. Others come from residents. Others come from tourists who describe the café they found down a side street or the tiny museum they happened to discover.
An AI system is then used to create hundreds of additional travel articles.
Those articles are polished. They are grammatically clean. They mention the Eiffel Tower, the Louvre, Montmartre, croissants and the Seine.
Another company scrapes the web and trains its own model on the accumulated content.
The new model produces even more articles.
A third system learns from those.
After several iterations, the internet contains thousands of articles about Paris, but many of them ultimately derive from the same small set of machine-generated patterns.
The resulting content may look perfectly plausible.
That’s precisely the problem.
There doesn’t need to be obvious nonsense.
The danger is that the diversity of information has quietly decreased.
You end up with 10,000 articles that sound different while communicating essentially the same handful of ideas.
The model has not learned Paris.
It has learned what previous models think a paragraph about Paris should look like.
We have already seen smaller versions of this problem
The phenomenon is not limited to language models.
Researchers studying image-generation systems have demonstrated related effects: when models are repeatedly trained using generated images, the resulting generations can increasingly lose characteristics of the original image distribution.
The effect can also be demonstrated with simpler statistical models.
That matters because model collapse isn’t simply a quirky defect of chatbots.
It is connected to a more fundamental issue:
If a learning system repeatedly learns from its own outputs, information can be lost.
The same intuition appears outside AI.
Think about photocopying a photocopy.
The first copy may look almost identical to the original.
Copy it again and you lose a little detail.
Repeat the process enough times and the result becomes increasingly distorted.
Model collapse is obviously more complicated than photocopying, but the analogy captures the feedback problem surprisingly well.
So is synthetic data the enemy?
No.
This is where discussions about model collapse sometimes become unnecessarily dramatic.
Synthetic data can be extremely useful.
It can help generate training examples that would be expensive or difficult to collect in the real world. It can be valuable for simulation, testing, data augmentation and specialized applications.
The important distinction is between using synthetic data and replacing reality with synthetic data.
Recent research has actually complicated the original model-collapse narrative in an important way.
A 2025 paper presented at ICML found that the outcome depends heavily on how synthetic and real data are managed. In experiments across several settings, replacing real data with successive generations of synthetic data produced collapse, while accumulating synthetic data alongside the available data could prevent the same kind of runaway degradation.
Another line of research has explored ways of producing synthetic or semi-synthetic text without creating the same destructive feedback loop, including modifying human-produced material rather than simply asking a model to generate an entirely new replacement dataset.
So the lesson isn’t:
“Never use AI-generated data.”
It is closer to:
“Don’t let the model become the only witness to reality.”
Solution #1: Keep the original data
This is probably the most straightforward defense.
If a company has access to high-quality human-generated training data today, it should not assume that the same data will remain recoverable from the internet tomorrow.
The original Nature research makes this point explicitly: preserving access to genuine human-produced data becomes increasingly important as machine-generated content becomes more prevalent online.
For an AI company, that could mean maintaining carefully curated datasets containing:
- human-written text
- original photographs
- recorded conversations
- domain-specific documents
- scientific material
- real-world measurements
- expert annotations
- human feedback
The principle is simple:
Keep a connection to the source.
A photograph of a forest is more valuable as evidence of the forest than ten AI-generated descriptions of what a forest should look like.
Solution #2: Track where training data came from
There is another practical challenge.
Once AI-generated content is everywhere, how do you know which content was written by a person and which was generated by a machine?
Get Pierre Schwartz’s stories in your inbox
Join Medium for free to get updates from this writer.
This is a data-provenance problem.
A future training pipeline may need to know not only:
“Is this document high quality?”
but also:
“Where did this document come from?”
That could involve metadata, provenance systems, dataset documentation, contractual information from data providers, or other mechanisms for distinguishing sources.
The goal isn’t necessarily to reject every synthetic example.
It is to prevent training pipelines from unknowingly treating billions of machine-generated documents as independent observations of human reality.
Solution #3: Don’t replace, accumulate
This may be one of the most interesting findings from recent research.
If each generation throws away the previous dataset and replaces it with newly generated material, the feedback loop becomes extremely aggressive.
But if data accumulate — preserving earlier information while adding synthetic examples — the dynamics can look very different.
Research presented at ICML in 2025 found that certain accumulation strategies can contain model collapse rather than allowing it to accelerate indefinitely.
This suggests a practical design principle:
Synthetic data should generally be an addition to the information ecosystem, not a replacement for it.
That distinction could become increasingly important as companies build models that continuously retrain on newly collected data.
Solution #4: Keep humans in the loop where the tails matter
Not every application needs the same level of protection.
If you’re generating thousands of slightly different backgrounds for a video game, losing some obscure part of the data distribution may not matter much.
But imagine training a model for medicine.
Or law.
Or cybersecurity.
Or scientific research.
Rare cases can be disproportionately important.
A medical dataset dominated by common conditions might look excellent statistically while being dangerously weak at recognizing unusual diseases.
The “tail” is not necessarily noise.
Sometimes the tail is where the important cases live.
That means human experts remain valuable precisely where a purely statistical approach may be tempted to discard unusual examples.
Solution #5: Evaluate against reality, not against the model
There is another trap here.
If Model B is trained on Model A’s outputs, and Model B is evaluated primarily using data produced by Model A, Model B can appear to perform remarkably well.
It has learned the benchmark.
But perhaps it hasn’t learned the world.
This is why independent evaluation sets matter.
For important systems, evaluation should ideally include data that were not generated by the same feedback loop used to train the model.
For example:
- newly collected human-written material
- real-world measurements
- expert-created test cases
- adversarial examples
- rare and unusual cases
- live operational data
A model should occasionally be asked to leave the comfortable world of synthetic benchmarks and face reality.
The bigger problem: the internet itself could become a feedback loop
This is what makes model collapse more interesting than an ordinary machine-learning failure.
AI companies don’t operate in isolation.
Their models produce content.
People publish that content.
Search engines index it.
Web crawlers collect it.
Datasets are assembled from the web.
New models train on those datasets.
Those models produce more content.
And the cycle repeats.
The internet could therefore become a gigantic self-referential training dataset.
The risk isn’t that every AI-generated article will be false.
It’s that the ratio between original observations and recycled interpretations could change over time.
That’s a much harder problem to see.
A completely nonsensical dataset would be easy to reject.
A dataset consisting of polished, plausible, repetitive information is much more dangerous.
There is an unexpected lesson here about human creativity
Model collapse also exposes something we sometimes forget about human-generated data.
Human beings are inconsistent.
We make mistakes.
We contradict each other.
We use strange expressions.
We tell stories badly.
We have regional accents.
We obsess over obscure topics.
We change our minds.
We describe the same event in completely different ways.
From the perspective of statistical compression, much of this looks like undesirable noise.
But that “noise” is also evidence of diversity.
It tells a model that the world is larger than the most common answer.
An AI trained entirely on polished AI outputs may become very good at producing the statistically expected response while becoming progressively worse at representing everything that falls outside that expectation.
In that sense, the messiness of human data is not merely an inconvenience.
It is part of the information.
The real-world rule of thumb
For organizations building or deploying AI systems, the practical lesson can be reduced to a few questions:
Where did this data come from?
How much of it is synthetic?
Has synthetic material replaced original observations?
Can we still access the original human-generated data?
Are we evaluating the model against independent, real-world examples?
Are rare cases being preserved?
These questions will become increasingly important as synthetic content becomes cheaper than human content.
And that economic incentive is worth paying attention to.
If generating 10,000 synthetic examples costs almost nothing while collecting 10,000 human examples costs months of work, organizations will naturally want to use the synthetic option.
The engineering challenge is making sure that efficiency doesn’t quietly destroy the diversity the system depends on.
Model collapse isn’t inevitable
This may be the most important conclusion.
The phrase “model collapse” can make the future sound predetermined: AI generates content, AI trains on that content, and eventually everything turns into gibberish.
The evidence is more nuanced.
The original research demonstrated that recursive training on generated data can cause severe degradation.
But subsequent work has shown that the outcome depends on the training strategy, including whether real and synthetic data are replaced, accumulated, filtered or otherwise managed.
That is good news.
It means model collapse is not simply a countdown to disaster.
It is a systems-design problem.
And systems-design problems can be managed.
The AI ecosystem needs a memory
Perhaps the best way to think about the issue is this:
AI needs a memory of where its information came from.
If every generation learns only from the previous generation, eventually we lose the connection to the original source.
But if we preserve human data, maintain provenance, use synthetic data deliberately, protect unusual examples and test models against reality, the feedback loop becomes much safer.
The irony is almost poetic.
We built generative AI to create an endless supply of information.
Now we may discover that information isn’t valuable simply because there is more of it.
Original information is valuable because it is original.
A million variations of the same idea are not necessarily a million new pieces of knowledge.
And as AI-generated content becomes a larger part of the internet, one of the most valuable resources in the world may become something surprisingly old-fashioned:
a record of what real people actually said, saw, measured, experienced and created.
The future of AI may depend not on generating more data, but on making sure we don’t lose the data that came first.
Further reading
- Shumailov et al., AI models collapse when trained on recursively generated data, Nature (2024).
- Gerstgrasser et al., Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data (2024).
- Kazdan et al., Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World, ICML/PMLR (2025).
- Zhu et al., How to Synthesize Text Data without Model Collapse?, ICML/PMLR (2025).
