From Rollouts to Recipes: Self-Contained Post-Training for LLMs

2026-09-01

AI Summary

提出Self-Routing框架，根据模型输出的行为状态选择优化方法，无需额外标注或采样，实验显示其在数学推理任务中优于其他方法。

Abstract

Post-training large language models usually applies a single training recipe to all samples, even though the model's own rollouts reveal different sample-level learning states. We propose Self-Routing, a behavior-conditioned post-training framework that uses rollout correctness and confidence to decide how each sample should be optimized. Depending on its behavior state, a sample is routed to GRPO, on-policy self-distillation, regularization, or skipping, allowing training to adapt without external teachers, extra annotations, or additional sampling. Experiments on mathematical reasoning across Qwen3 and Qwen3.5 backbones show that Self-Routing consistently improves over uniform GRPO, uniform OPSD, fixed mixtures, and simpler routing baselines. Further analyses show that the routing distribution changes over training and reduces unnecessary updates on low-signal or already stable samples.

Problem

• post-training
• large language models
• sample-level learning states
• rollout correctness
• confidence

Innovation

• Self-Routing
• behavior-conditioned post-training
• rollout correctness and confidence
• adaptive optimization
• without external teachers