ReMiT: RL-Guided Mid-Training for Iterative LLM Evolution

2026-02-03

AI Summary

This work addresses the limitation of conventional large language model training pipelines, which are unidirectional and lack feedback from post-training to pre-training, thereby hindering continuous model evolution. The authors propose an iterative self-augmentation training framework that integrates reinforcement learning during the pre-training annealing phase to dynamically reweight tokens relevant to reasoning. This approach establishes a teacher- and reference-model-free bidirectional training loop, enabling post-training signals to inform and refine pre-training. Evaluated across ten benchmarks spanning mathematical reasoning, code generation, and general-purpose reasoning, the method achieves an average performance gain of 3% and sustains over 2% improvement in subsequent post-training stages, marking the first demonstration of reasoning-driven co-optimization between pre-training and post-training.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Standard training pipelines for large language models (LLMs) are typically unidirectional, progressing from pre-training to post-training. However, the potential for a bidirectional process--where insights from post-training retroactively improve the pre-trained foundation--remains unexplored. We aim to establish a self-reinforcing flywheel: a cycle in which reinforcement learning (RL)-tuned model strengthens the base model, which in turn enhances subsequent post-training performance, requiring no specially trained teacher or reference model. To realize this, we analyze training dynamics and identify the mid-training (annealing) phase as a critical turning point for model capabilities. This phase typically occurs at the end of pre-training, utilizing high-quality corpora under a rapidly decaying learning rate. Building upon this insight, we introduce ReMiT (Reinforcement Learning-Guided Mid-Training). Specifically, ReMiT leverages the reasoning priors of RL-tuned models to dynamically reweight tokens during the mid-training phase, prioritizing those pivotal for reasoning. Empirically, ReMiT achieves an average improvement of 3\% on 10 pre-training benchmarks, spanning math, code, and general reasoning, and sustains these gains by over 2\% throughout the post-training pipeline. These results validate an iterative feedback loop, enabling continuous and self-reinforcing evolution of LLMs.

Problem

• large language models
• iterative evolution
• bidirectional training
• reinforcement learning
• mid-training

Innovation

• Reinforcement Learning
• Mid-Training
• Iterative LLM Evolution
• Token Reweighting
• Self-Reinforcing Training