Knowledge Engine

Do AI models decay when they learn from each other?

·no read data

The idea

Generative models such as large language modelslarge language model A computer program trained on very large amounts of text to predict and produce written language. learn from text and images gathered at scale, much of it from the web. As those models produce more of what is online, later models will increasingly be trained on their predecessors’ output. Ilia Shumailov and colleagues, in Nature in 2024, asked what happens when that loop repeats.

Their answer was that indiscriminate training on model-generated content causes what they called model collapsemodel collapse A gradual loss of variety and accuracy in AI models that are trained, round after round, on output produced by earlier models.. They describe two stages. In early collapse, the model begins losing information about the tails of the distribution: the rare words, unusual cases and minority patterns. In late collapse, the model converges on something that bears little resemblance to the original data, often with much less variety. They attribute this mainly to statistical approximation error: each generation sees only a finite sample, so rare events are likely to be missed, and once missed they cannot be learned back. Limits on what a model can represent, and on how it is trained, add further error.

A concrete version: suppose a word appears once in a million sentences. A model trained on a modest sample may never see it and so never produce it. The next model, trained on the first model’s output, has no chance of learning it at all. Repeat this and the distribution narrows round by round.

They demonstrated the effect in simple Gaussian mixture models, in variational autoencodersvariational autoencoder A type of AI model that learns to squeeze examples into a compact summary and rebuild new, similar examples from it., and in a language model, OPT-125m, fine-tunedfine-tuning Further training of an already trained AI model on a smaller, chosen set of examples to change how it behaves. on the wikitext2 dataset over several generations.

Why it might change a decision

The two studies discussed here make a sharp, testable distinction. If each new model is trained only on the previous model’s output, quality should keep falling. If the original human data is kept and synthetic datasynthetic data Data produced by a computer model rather than collected from real people or events. is added on top, quality should degrade much more slowly or level off. That changes what matters in practice: not just whether synthetic data is present, but whether the real data is retained alongside it.

How solid is this?

Both papers are controlled experiments on small models over a limited number of generations, supported by theory on idealised models. Neither observes collapse in deployed commercial systems.

Shumailov and colleagues fine-tuned a pre-trained model rather than training from scratch, and say larger models were not tested because of cost. Their theory concerns many generations, while experiments ran only a few; they note full collapse may or may not appear within a small number of steps. They also found that keeping a random 10% of the original data in each generation led to only minor degradation. A correction to the paper was published in 2025.

Matthias Gerstgrasser and colleagues, in a 2024 preprintpreprint A research paper shared publicly before, or without, formal peer review by a journal., argued that the main set-up behind alarming results was data replacement, where each round’s synthetic data stands in for the real data. They pretrained small language models (GPT-2 at 9 million parametersparameter One of the adjustable numbers inside an AI model that training tunes; model size is usually counted in them. and Llama 2 models up to about 125 million) on the TinyStories dataset. Replacing data raised test error as iterations went on; accumulating synthetic data alongside the real data gave equal or lower error. They found similar patterns in diffusion models for molecules and in VAEs for images. In a linear regression model they proved that under accumulation the error stays below a fixed bound however many rounds pass, whereas under replacement it grows steadily.

Their own caveats matter. Accumulation trains on ever larger datasets; when they matched the amount of compute, performance still degraded, though more slowly. For VAEs, error under accumulation still rose with iterations. They note an earlier study that found fast deterioration even with accumulation and leave the conflict unresolved. Their conclusion is that recursion “may not be as dire as had been portrayed”, not that the problem is solved.

The popular claim that AI will inevitably poison itself outruns these data. So does the reverse claim that accumulation makes the concern irrelevant. What is supported is narrower: repeated training on synthetic data alone loses rare information, and keeping real data in the mix substantially reduces, but does not always eliminate, the loss in small models. Whether the real web behaves more like replacement or accumulation is an open question neither paper measured.

Sources

  • Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., & Gal, Y. (2024). AI models collapse when trained on recursively generated data. Nature, 631, 755-759. doi:10.1038/s41586-024-07566-y
  • Gerstgrasser, M., Schaeffer, R., Dey, A., Rafailov, R., Sleight, H., Hughes, J., Korbak, T., Agrawal, R., Pai, D., Gromov, A., Roberts, D. A., Yang, D., Donoho, D. L., & Koyejo, S. (2024). Is model collapse inevitable? Breaking the curse of recursion by accumulating real and synthetic data. arXiv preprint. arXiv:2404.01413

Check yourself

Answers are checked in your browser. Nothing is recorded or sent anywhere.

  1. According to Shumailov and colleagues, what is lost first in early model collapse?

  2. What is the key difference between the 'replace' and 'accumulate' set-ups that Gerstgrasser and colleagues compared?

  3. What did Shumailov and colleagues report when 10% of the original data was kept in each generation of their language model experiment?

  4. Which is the fairest description of the evidence reviewed here?

Questions without answers

  1. The web now contains a growing share of machine-written text that is not labelled as such. Is the real web closer to the 'replace' or the 'accumulate' condition, and what would you need to measure to find out?
  2. Collapse first removes rare cases. In a field you know, which rare but important cases would be hardest to notice going missing from a model's output?

Back to the archive