Repeated post-training is not Self-improving: Diagnosing Scientific Amnesia in Continual DPO Pipelines

2026-06-17

AI Summary

This work addresses the “scientific amnesia” problem in continual DPO training, where models retain prior behaviors yet fail to accumulate reusable methodological knowledge. We formalize this phenomenon as a measurable, industrial-scale challenge and introduce a comprehensive diagnostic framework comprising a procedural pipeline, a 30-round HumanEval subdomain benchmark, and five categories of strategy proposers. Leveraging chain-wise training of Qwen2.5-7B-Instruct with FSDP-sharded DPO checkpoints, we evaluate strategies including rule-based scheduling, retrieval-augmented memory, Bayesian optimization, and a meta-scientific reasoning model (MSCL). Experiments reveal that only rule-based scheduling improves performance in homogeneous single-seed chains, while MSCL uniquely enhances outcomes in heterogeneous chains. Across multi-seed settings, retrieval-augmented memory yields the highest average gains, though differences among methods do not reach statistical significance.

Abstract

Industrial LLM teams often ship behavior updates by repeatedly DPO-training a base model on sequences of related preference-data campaigns. The dominant failure mode in this regime is not always classical catastrophic forgetting: a pipeline may preserve previously learned behaviors while still failing to accumulate reusable methodological knowledge about how to train the next campaign. We call this failure mode scientific amnesia. This paper turns that practitioner intuition into a measurable industrial problem. We contribute: (i) a diagnostic suite for amnesia, (ii) a Program-based pipeline that chains FSDP-sharded DPO checkpoints across Qwen2.5-7B-Instruct runs, (iii) a 30-campaign HumanEval subdomain benchmark, and (iv) a comparative diagnostic study of five strategy proposers: random memory, rule-based scheduling, retrieval-only memory, warm-start Bayesian optimization, and MSCL, a meta-scientific memory and reasoner candidate. Across a single-seed 5-condition * 3-step real-LM chain, 4 of 5 candidates degrade in step-level peak pass@1, including MSCL; only the deliberately conservative rule-based schedule improves. Follow-up pilots qualify rather than overturn this finding: in a heterogeneous chain, MSCL is the only completed candidate that improves, whereas in a small multi-seed homogeneous sweep, retrieval-only has the best mean Delta and no pairwise candidate gap is statistically distinguishable. The contribution is therefore diagnostic, not a claim that MSCL solves the problem: scientific amnesia is observable in a production-like continual-DPO pipeline, and conclusions about interventions depend sharply on chain regime, evaluator design, and seed coverage.

Problem

• scientific amnesia
• continual DPO
• preference optimization
• methodological knowledge
• catastrophic forgetting

Innovation

• scientific amnesia
• continual DPO
• diagnostic suite
• meta-scientific reasoning
• preference optimization