Can LLMs Learn to Reason Robustly under Noisy Supervision?

2026-04-05

AI Summary

This work addresses the vulnerability of large language models in reinforcement learning to noisy labels caused by scarce expert annotations, a challenge inadequately handled by existing methods. The study is the first to identify two distinct mechanisms through which label noise operates in reinforcement learning: inactive and active. To mitigate this issue, the authors propose an online label refinement strategy, OLR, which dynamically corrects labels by integrating rollout pass-rate slope and historical prediction consistency, further enhanced with majority voting and policy stability checks. Evaluated across six mathematical reasoning benchmarks and three out-of-distribution tasks under label noise ratios ranging from 0.1 to 0.9, OLR consistently improves average performance by 3.3%–4.6%, demonstrating significantly enhanced model robustness.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) effectively trains reasoning models that rely on abundant perfect labels, but its vulnerability to unavoidable noisy labels due to expert scarcity remains critically underexplored. In this work, we take the first step toward a systematic analysis of noisy label mechanisms in RLVR. In contrast to supervised classification, most RLVR algorithms incorporate a rollout-based condition: a label's influence on training is contingent on whether the current policy can generate rollouts that realize it, a property that naturally extends to noisy labels. Based on this observation, we distinguish two types of noise: inactive noisy labels, which reduce data efficiency, and active noisy labels, which are reinforced and risk skewing the model toward incorrect distributions. From experiments on training with noisy samples, we identify an Early Correctness Coherence phenomenon: although noisy samples begin to lag behind in later stages, accuracy on both clean and noisy samples increases similarly in early training. Motivated by this dynamic, we propose Online Label Refinement (OLR), which progressively corrects potentially noisy labels with majority-voted answers when two conditions hold: a positive slope in the majority answer's rollout pass rate and stable historical consistency across updates, enabling gradual self-correction as the policy improves. We evaluate OLR on six in-distribution mathematical reasoning benchmarks (AIME24/25, AMC, MATH-500, Minerva, and Olympiad) and three out-of-distribution tasks (ARC-c, GPQA-diamond, and MMLU-pro). Across noise ratios from 0.1 to 0.9, OLR consistently improves robustness under both inactive and active noisy-label settings, achieving average gains of 3.6% to 3.9% on in-distribution benchmarks and 3.3% to 4.6% on out-of-distribution evaluations.

Problem

• noisy supervision
• robust reasoning
• LLMs
• RLVR
• label noise

Innovation

• noisy supervision
• reinforcement learning with verifiable rewards
• online label refinement
• early correctness coherence
• self-correction