STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning

2026-05-13

AI Summary

Current large language model reasoning training relies on scalar rewards or frozen critics, suffering from limited information bandwidth and constrained policy improvement. This work proposes STRIDE, a novel framework that introduces a learnable stepwise language feedback mechanism, jointly training a generator and a generative verifier to redirect intermediate steps of reasoning trajectories using only end-to-end outcome rewards. The approach requires no external annotations, enables harm-controllable continuous policy optimization, and effectively alleviates learning failures in zero-pass-rate scenarios. Evaluated across multiple reasoning benchmarks, STRIDE significantly outperforms state-of-the-art methods, achieving breakthrough performance especially on tasks where scalar rewards prove inadequate.

Abstract

Recent advances in Reinforcement Learning (RL) have underscored its potential for incentivizing reasoning capabilities of Large Language Models (LLMs). However, existing step-level efforts suffer from costly annotations that limit domain coverage, while scalar scores further impose an information bottleneck, offering insufficient semantic bandwidth to improve intermediate decisions. Alternative language-critique approaches, which rely on frozen or external critics, provide richer textual feedback but lack the scalability needed for sustained policy improvement. In this work, we propose language-driven stepwise trajectory redirection, termed as STRIDE, a novel training framework that shifts process supervision from scalar rewards to learnable stepwise language feedback. Specifically, we co-train a generator and a generative verifier using only outcome-based rewards, eliminating external annotations, while delivering sustained policy improvement through jointly aligned verifier training. The verifier's stepwise language critiques explicitly localize and explain failures, enabling the generator to redirect reasoning trajectories at intermediate steps toward alternative decisions. The trajectory redirection design guarantees harmless policy improvement, even under noisy or suboptimal verifier feedback. Experiments on diverse reasoning benchmarks show that STRIDE significantly outperforms state-of-the-art baselines, as well as achieving breakthroughs on zero-pass-rate problems where scalar methods yield no learning signal in our ablation studies, demonstrating the effectiveness of learnable stepwise language feedback for enhancing LLM reasoning.

Problem

• Reinforcement Learning
• Large Language Models
• Process Supervision
• Language Feedback
• Reasoning

Innovation

• learnable stepwise feedback
• language-driven reasoning
• generative verifier
• trajectory redirection
• process supervision