Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence

2025-10-18

AI Summary

This work addresses model collapse—a critical failure mode in iterative self-training of generative models using synthetic data. To mitigate this, we propose an external validation mechanism that filters synthetic data using an oracle (e.g., human annotators or a stronger model). Theoretical analysis is conducted within a linear regression framework, and empirical evaluation employs variational autoencoders (VAEs) on benchmarks including MNIST. Results demonstrate that the validation mechanism effectively prevents collapse and steers model convergence toward a “knowledge center” defined by the validator’s representation. It yields significant short-term performance gains while ensuring stable long-term convergence. Crucially, this study provides the first dual theoretical and experimental evidence that external validation of synthetic data is a necessary intervention to avert model collapse. Moreover, it reveals an inherent trade-off between immediate performance improvement and the asymptotic convergence limit—highlighting fundamental constraints in self-training dynamics.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Synthetic data has been increasingly used to train frontier generative models. However, recent study raises key concerns that iteratively retraining a generative model on its self-generated synthetic data may keep deteriorating model performance, a phenomenon often coined model collapse. In this paper, we investigate ways to modify this synthetic retraining process to avoid model collapse, and even possibly help reverse the trend from collapse to improvement. Our key finding is that by injecting information through an external synthetic data verifier, whether a human or a better model, synthetic retraining will not cause model collapse. To develop principled understandings of the above insight, we situate our analysis in the foundational linear regression setting, showing that iterative retraining with verified synthetic data can yield near-term improvements but ultimately drives the parameter estimate to the verifier's "knowledge center" in the long run. Our theory hence predicts that, unless the verifier is perfectly reliable, the early gains will plateau and may even reverse. Indeed, these theoretical insights are further confirmed by our experiments on both linear regression as well as Variational Autoencoders (VAEs) trained on MNIST data.

Problem

• Preventing model collapse in generative models using synthetic data verification
• Investigating synthetic retraining modifications to reverse performance deterioration
• Analyzing long-term convergence to verifier's knowledge center in iterative training

Innovation

• Uses synthetic data verifier to prevent collapse
• Verifier injection enables long-term parameter convergence
• Applies verification to both linear and VAE models