Text2Grad: Reinforcement Learning from Natural Language Feedback

2025-05-28

AI Summary

Traditional RLHF relies on coarse-grained scalar rewards, failing to reveal fine-grained causes of generation failures, resulting in inefficient and uninterpretable policy optimization. To address this, we propose SpanRL—the first end-to-end framework that directly maps free-text feedback into token-level differentiable gradients for span-level local policy updates. Our method comprises three components: (1) a feedback-token alignment pipeline; (2) joint modeling of interpretable critiques and span-level rewards; and (3) a natural-language-gradient-based backpropagation optimizer. Evaluated on summarization, code generation, and question answering, SpanRL significantly outperforms scalar RLHF and prompt-augmented baselines. Crucially, it delivers strong, per-span attributional interpretability—enabling, for the first time, parameter-level, localized, and differentiable policy correction driven entirely by textual feedback.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Traditional RLHF optimizes language models with coarse, scalar rewards that mask the fine-grained reasons behind success or failure, leading to slow and opaque learning. Recent work augments RL with textual critiques through prompting or reflection, improving interpretability but leaving model parameters untouched. We introduce Text2Grad, a reinforcement-learning paradigm that turns free-form textual feedback into span-level gradients. Given human (or programmatic) critiques, Text2Grad aligns each feedback phrase with the relevant token spans, converts these alignments into differentiable reward signals, and performs gradient updates that directly refine the offending portions of the model's policy. This yields precise, feedback-conditioned adjustments instead of global nudges. Text2Grad is realized through three components: (1) a high-quality feedback-annotation pipeline that pairs critiques with token spans; (2) a fine-grained reward model that predicts span-level reward on answer while generating explanatory critiques; and (3) a span-level policy optimizer that back-propagates natural-language gradients. Across summarization, code generation, and question answering, Text2Grad consistently surpasses scalar-reward RL and prompt-only baselines, providing both higher task metrics and richer interpretability. Our results demonstrate that natural-language feedback, when converted to gradients, is a powerful signal for fine-grained policy optimization. The code for our method is available at https://github.com/microsoft/Text2Grad

Problem

• Converts free-form textual feedback into span-level gradients for precise model adjustments
• Aligns feedback phrases with token spans to generate differentiable reward signals
• Improves RLHF by replacing coarse scalar rewards with fine-grained natural-language gradients

Innovation

• Converts textual feedback into span-level gradients
• Aligns feedback phrases with relevant token spans
• Performs gradient updates on specific policy portions