DenoiseRL: Bootstrapping Reasoning Models to Recover from Noisy Prefixes

2026-05-27

AI Summary

This work addresses the scalability limitations of existing reinforcement learning (RL) approaches for enhancing large language models’ reasoning capabilities, which typically rely on strong teacher models or high-quality, challenging datasets. The authors propose DenoiseRL, a novel framework that leverages the model’s own erroneous reasoning trajectories as training signals. Through a recovery-oriented RL mechanism, DenoiseRL transforms noisy reasoning prefixes into opportunities for improvement, enabling self-supervised training without external supervision. This approach significantly improves exploration efficiency and scalability, consistently outperforming strong on-policy RL baselines across mathematical and general reasoning benchmarks. Notably, as task difficulty increases, DenoiseRL demonstrates enhanced self-correction capabilities and training efficiency, highlighting its robustness in complex reasoning scenarios.

Abstract

Reinforcement learning has become a central paradigm for advancing reasoning in large language models, yet most existing methods still depend on stronger teacher models or heavily curated difficult datasets, limiting scalable capability improvement. In this paper, we introduce DenoiseRL, a reinforcement learning framework that substitutes external supervision with recovery-oriented optimization over failures from weak models. Instead of relying on stronger supervision or carefully engineered data, DenoiseRL learns directly from incorrect reasoning traces by converting them into opportunities for improvement, making training more scalable and less dependent on external resources. This yields a richer and more diverse learning signal, improving exploration efficiency from imperfect model behavior. As a result, DenoiseRL improves reasoning performance and overall training efficiency while reducing the need for expensive data curation or stronger teacher models. Empirically, DenoiseRL consistently outperforms strong on-policy RL baselines across competitive mathematical and general reasoning benchmarks and promotes stronger self-corrective behavior as training difficulty increases, highlighting an effective and scalable alternative pathway for improving reasoning in large language models.

Problem

• reinforcement learning
• reasoning models
• noisy prefixes
• scalable improvement
• external supervision

Innovation

• DenoiseRL
• reinforcement learning
• self-correction
• noisy reasoning traces
• scalable training