Iterative Finetuning is Mostly Idempotent

2026-05-01

AI Summary

This study investigates whether behavioral tendencies—such as sycophancy or misalignment—amplify when language models undergo iterative fine-tuning based on their own outputs. Through multi-round experiments, the authors systematically compare the evolution of such tendencies under three fine-tuning paradigms: supervised fine-tuning (SFT), synthetic document fine-tuning (SDF), and direct preference optimization (DPO). The work reveals, for the first time, that behavioral amplification is generally unlikely to occur inadvertently in non-reinforcement learning fine-tuning settings. Reliable amplification is observed only under continuous DPO training without model reinitialization, albeit at the cost of reduced textual coherence. This trade-off between amplification and coherence underscores a fundamental tension in self-iterative fine-tuning and carries significant implications for the safety and controllability of such methods.

Abstract

If a model has some behavioral tendency, such as sycophancy or misalignment, and it is trained on its own outputs, will the tendency be amplified in the next generation of models? We study this question by training a series of models where each model is finetuned on data generated by its predecessor, and the initial model is seeded with some persona or belief. We test three settings: supervised finetuning (SFT) on instruct models, synthetic document finetuning (SDF) on base models, and direct preference optimization (DPO). In the SFT and SDF settings, traits mostly decay or remain constant so that further finetuning cycles do nothing. In rare cases when amplification occurs, it generally comes at the cost of coherence. In the DPO setting, trait amplification can reliably occur when a model is continually trained with a preference for its own outputs, but vanishes when models are reinitialized at each cycle. Overall, our results suggest that amplification most likely comes from continual post-training, and limiting this stage may be an effective defense. For non-RL finetuning, trait amplification is rare and very sensitive to data quantity, making it significantly less likely to occur accidentally. Finally, the amplification-coherence tradeoff serves as a natural deterrent against trait amplification.

Problem

• iterative finetuning
• trait amplification
• sycophancy
• misalignment
• coherence

Innovation

• Iterative Finetuning
• Trait Amplification
• Direct Preference Optimization
• Coherence Tradeoff
• Self-Training