Annotations Mitigate Post-Training Mode Collapse

2026-05-11

AI Summary

This work addresses the semantic mode collapse induced by post-training methods such as supervised fine-tuning, which degrades the high-entropy diversity acquired during pretraining—a problem that intensifies with increasing model scale. To mitigate this, the authors propose Annotation-Anchored Training: during pretraining, rich semantic annotations are incorporated to construct a diverse distribution, which is then anchored throughout post-training; at inference, generation is guided by sampling from these diverse annotations. By leveraging semantic annotations as a bridge, this approach uniquely preserves preference alignment while substantially alleviating mode collapse. Experimental results demonstrate that, compared to standard supervised fine-tuning, the proposed method reduces diversity loss to approximately one-sixth, with consistent performance gains scaling favorably with model size.

Abstract

Post-training (via supervised fine-tuning) improves instruction-following, but often induces semantic mode collapse by biasing models toward low-entropy fine-tuning data at the expense of the high-entropy pretraining distribution. Crucially, we find this trade-off worsens with scale. To close this semantic diversity gap, we propose annotation-anchored training, a principled method that enables models to adopt the preference-following behaviors of post-training without sacrificing the inherent diversity of pretraining. Our approach is simple: we pretrain on documents paired with semantic annotations, inducing a rich annotation distribution that reflects the full breadth of pretraining data, and we preserve this distribution during post-training. This lets us sample diverse annotations at inference time and use them as anchors to guide generation, effectively transferring pretraining's semantic richness into post-trained models. We find that models trained with annotation-anchored training can attain $6 \times$ less diversity collapse than models trained with SFT, and improve with scale.

Problem

• post-training
• mode collapse
• semantic diversity
• supervised fine-tuning
• pretraining distribution

Innovation

• annotation-anchored training
• semantic mode collapse
• post-training
• semantic diversity
• supervised fine-tuning