The Models Are Already Eating Themselves

Train an AI on human text. Train the next generation on that AI's outputs. Repeat. The 2024 Nature paper on this process isn't speculative. The mechanism is mathematical, the effect is irreversible, and the web passed the threshold for triggering it somewhere around mid-2024.

This is distinct from the general worry that "AI content is low quality." A single generation of synthetic data in a training corpus is usually harmless — sometimes beneficial, when it's high-quality targeted synthetic augmentation. The specific failure mode Ilia Shumailov and colleagues proved in Nature is recursive training: each new model trained partly on outputs from the previous generation. As AI-generated text flooded the web through 2024 and 2025, that loop became the default operating condition, not an edge case.

The Mechanism in the Nature Paper

The paper — "AI models collapse when trained on recursively generated data," Nature 631, 755–759 (July 2024) — ran three model classes through multi-generational training: large language models, variational autoencoders, and Gaussian mixture models. All three collapsed. The failure had two distinct stages.

Early collapse: The tails of the distribution disappear. Minority data — rare words, unusual phrasings, low-frequency knowledge — gets underrepresented as each successive model rounds toward the statistical mean of what came before. The model doesn't know these things are gone. It simply stops producing them.

Late collapse: Variance shrinks toward zero. The model converges on a narrow band of outputs. Different inputs produce nearly identical results. The model has memorized the center of the distribution and lost access to everything outside it.

The Cambridge extension of this work — the "curse of recursion" paper — added a finding almost no coverage mentions: early collapse happens even with zero function estimation error. Train the model perfectly and you still lose your tails. It's not a quality-of-training problem. It's a mathematical property of recursive information compression. You cannot engineer your way out of it without addressing the data source itself.

What the Web Looks Like Right Now

By the end of 2024, approximately 57% of new English-language text published online was written with significant AI assistance, according to analysis from the Tollens Institute. This is not a controlled synthetic corpus. This is the training data for next-generation models, indistinguishably mixed with human writing and crawled at web scale.

Any model trained on an unfiltered web scrape from 2025 onward is ingesting recursive synthetic data by default. The Shumailov mechanism applies. The first generation after this threshold may not show measurable collapse. The third will.

The counterargument is that labs filter their training corpora — deduplicate, quality-score, exclude known AI-generated domains. This is true in theory and incomplete in practice. Quality filters catch obvious low-effort content. They don't catch fluent, well-structured AI text that happens to represent a narrowed distribution. The collapse mechanism doesn't require bad writing. It requires recursive statistical compression, which fluent writing performs just as efficiently as poor writing.

Why Benchmarks Won't Catch It

Standard capability benchmarks test whether a model can answer questions correctly. Model collapse doesn't make models wrong. It makes them narrow. A collapsed model will score well on MMLU. It will fail on anything requiring unusual knowledge, edge-case reasoning, or information from the long tail of human experience and expertise.

This compounds a problem documented elsewhere: AI Benchmarks Have Become Marketing Documents covers how current benchmarks measure the center of the distribution, not the tails. Shumailov's collapse mechanism specifically attacks the tails. The benchmark system is blind to the part of capability that collapse destroys first.

The practical implication: models could get worse at the things that matter most — rare disease diagnosis, obscure legal precedent, specialized domain knowledge — while continuing to improve on benchmarks that test common knowledge. The scores go up. The capability goes down in the places where it matters most.

Who Is Actually Working on This

Three mitigation strategies exist. Data provenance tracking: knowing which data in a training corpus is synthetic and excluding it from recursive inclusion. Watermarking: tagging AI-generated content at generation time so it can be identified and filtered. Human-only sets: maintaining training corpora composed exclusively of human-generated text, separate from the web crawl.

None of these operate at scale. Data provenance across a 100-billion-token corpus requires both technical infrastructure and cooperation from every source that data came from. Watermarking requires universal adoption across every major AI provider and is trivially stripped by post-processing. Human-only sets exist — academic papers, curated archives, pre-2023 text — but they're finite and non-growing.

The major AI labs have internal research programs on this problem. None have made public commitments to resolving it before the next training run. The Shumailov paper received 800+ citations in its first year. The operational response to it has been private and partial.

The Timeline the Math Implies

Shumailov's model projects collapse over multiple training generations, not a single one. With annual model refresh cycles at the leading labs, multi-generational collapse becomes empirically visible within three to five years of the recursive data flood's onset — roughly 2027 to 2029, given the 2024 to 2025 content surge.

This is not a prediction. It is what the demonstrated mechanism implies under realistic operating conditions. The distribution tails are already narrowing. The variance compression will follow. The question is not whether to take model collapse seriously. It is whether the infrastructure to address it gets built before the damage is visible in deployed systems — or only after.

The models eating themselves today are the training signal for tomorrow's models. That feedback loop is already running. The math doesn't wait for a policy decision.

Photo: Google DeepMind / Pexels