Language Generation with Replay: A Learning-Theoretic View of Model Collapse

2026-03-12

AI Summary

This work addresses the problem of model collapse in large language models caused by training on data contaminated with their own generated content. From a learning-theoretic perspective, the authors propose a replay adversary framework that injects historical model outputs into the training stream, offering the first fine-grained theoretical characterization of model collapse within the language generation limit paradigm. By integrating language generation limit theory, replay adversarial modeling, and non-uniform generation analysis, they reveal that replay is benign under strong consistency generation but induces performance degradation under weak generation paradigms. This study not only elucidates the effectiveness and limitations of existing mitigation strategies—such as data filtering, watermarking, and output curation—but also provides rigorous theoretical grounding and delineates their failure boundaries.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: (Large) Language Models
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

As scaling laws push the training of frontier large language models (LLMs) toward ever-growing data requirements, training pipelines are approaching a regime where much of the publicly available online text may be consumed. At the same time, widespread LLM usage increases the volume of machine-generated content on the web; together, these trends raise the likelihood of generated text re-entering future training corpora, increasing the associated risk of performance degradation often called model collapse. In practice, model developers address this concern through data cleaning, watermarking, synthetic-data policies, or, in some cases, blissful ignorance. However, the problem of model collapse in generative models has not been examined from a learning-theoretic perspective: we study it through the theoretical lens of the language generation in the limit framework, introducing a replay adversary that augments the example stream with the generator's own past outputs. Our main contribution is a fine-grained learning-theoretic characterization of when replay fundamentally limits generation: while replay is benign for the strongest notion of uniform generation, it provably creates separations for the weaker notions of non-uniform generation and generation in the limit. Interestingly, our positive results mirror heuristics widely used in practice, such as data cleaning, watermarking, and output filtering, while our separations show when these ideas can fail.

Problem

• model collapse
• language generation
• replay
• learning theory
• training data contamination

Innovation

• model collapse
• language generation
• learning theory
• replay adversary
• generation in the limit