Training Reasoning Models on Saturated Problems via Failure-Prefix Conditioning

2026-01-28

AI Summary

This work addresses the performance saturation commonly observed in reinforcement learning for reasoning tasks, which stems from the difficulty of sampling informative failure trajectories. To overcome this limitation, the authors propose a failure-prefix conditioning mechanism that identifies and reuses prefixes of rare erroneous reasoning paths to guide the model toward error-prone states, thereby uncovering latent learning signals. Integrated within the RLVR framework, this approach combines targeted error-trajectory sampling with an iterative prefix-refreshing strategy. It achieves significant performance gains on saturated problems while maintaining token efficiency and enhances robustness against misleading prefixes, yielding improvements comparable to those obtained through training on medium-difficulty tasks.

Technology Category

• Machine Learning: Reinforcement Learning
• Knowledge Representation and Reasoning: Reasoning with Beliefs
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved the reasoning abilities of large language models (LLMs), yet training often stalls as problems become saturated. We identify the core challenge as the poor accessibility of informative failures: learning signals exist but are rarely encountered during standard rollouts. To address this, we propose failure-prefix conditioning, a simple and effective method for learning from saturated problems. Rather than starting from the original question, our approach reallocates exploration by conditioning training on prefixes derived from rare incorrect reasoning trajectories, thereby exposing the model to failure-prone states. We observe that failure-prefix conditioning yields performance gains matching those of training on medium-difficulty problems, while preserving token efficiency. Furthermore, we analyze the model's robustness, finding that our method reduces performance degradation under misleading failure prefixes, albeit with a mild trade-off in adherence to correct early reasoning. Finally, we demonstrate that an iterative approach, which refreshes failure prefixes during training, unlocks additional gains after performance plateaus. Overall, our results suggest that failure-prefix conditioning offers an effective pathway to extend RLVR training on saturated problems.

Problem

• reasoning models
• saturated problems
• failure trajectories
• reinforcement learning
• training stagnation

Innovation

• failure-prefix conditioning
• reasoning models
• saturated problems
• RLVR
• reinforcement learning