Curated Synthetic Data Doesn't Have to Collapse: A Theoretical Study of Generative Retraining with Pluralistic Preferences

2026-05-08

AI Summary

This work addresses the collapse of output diversity in recursively retrained generative models that rely solely on a single reward signal. From an alignment perspective, the authors propose a data filtering mechanism based on multiple reward functions to preserve diversity throughout the retraining process. By leveraging game-theoretic and probabilistic distributional dynamics, they formally characterize recursive training under multiple rewards for the first time and theoretically prove that, under heterogeneous preferences, the model converges to a stable distribution corresponding to the weighted Nash bargaining solution. This condition guarantees the retention of diversity within high-reward regions and enables diverse, stable synthetic data retraining without requiring access to real data.

Abstract

Recursive retraining of generative models poses a critical representation challenge: when synthetic outputs are curated based on a fixed reward signal, the model tends to collapse onto a narrow set of outputs that over-optimize that objective. Prior work suggests that such collapse is unavoidable without adding real data into the mix. We revisit this conclusion from an alignment perspective and show that collapse can be mitigated through curation based on multiple reward functions. We formalize the dynamics of recursive training under heterogeneous preferences and prove that, under certain conditions, the model converges to a stable distribution that allocates probability mass across competing high-reward regions. The limiting distribution preserves diversity and provably satisfies a weighted Nash bargaining solution, offering a formal interpretation of value aggregation in synthetic retraining loops.

Problem

• generative retraining
• model collapse
• synthetic data
• reward functions
• distribution diversity

Innovation

• generative retraining
• pluralistic preferences
• reward diversity
• Nash bargaining
• synthetic data collapse