Model Collapse

Model collapse occurs when a model is trained on data that was itself generated by a model, and this process repeats across generations. Each generation of training amplifies small errors and biases from the previous one while losing rare but valid patterns from the original distribution. Over successive generations, the model's outputs converge toward a narrower and less representative subset of what real-world data actually looks like.

The mechanism is straightforward. A model trained on real data approximates the true distribution but inevitably makes small errors, overrepresenting common patterns and underrepresenting rare ones. When the next model trains on that output instead of real data, it treats those errors as ground truth. The overrepresented patterns become even more dominant. The underrepresented ones fade further. After several generations, the model produces outputs that are internally consistent but bear decreasing resemblance to the original distribution.

Model collapse has become a pressing concern as the internet fills with AI-generated content. Models trained on web-scraped data increasingly encounter synthetic text, images, and code produced by earlier models. Without careful data curation to maintain the proportion of human-generated content, future models risk training on a feedback loop of their predecessors' outputs. Research has shown that even a small fraction of synthetic data in the training set can measurably degrade model quality over multiple training generations, making data provenance and filtering among the most important problems in modern AI development.