Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoning

2026-02-02

AI Summary

This work addresses the limitation of existing LLM-as-a-Judge reinforcement learning approaches, which provide only sparse sequence-level rewards and thus struggle to support fine-grained optimization for complex, long-horizon reasoning in open-domain tasks. To overcome this, the authors propose Grad2Reward, a novel framework that leverages gradient information from a single backward pass of the Judge model to derive dense, token-level process rewards, enabling precise credit assignment. Additionally, Grad2Reward introduces a self-critique mechanism that allows the policy model to perform efficient self-supervised optimization using its own evaluative signals, eliminating the need for external reward models or stronger Judges. Experiments demonstrate that the method significantly improves both reasoning quality and training efficiency across diverse open-domain tasks, validating its effectiveness and generalization capability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed significant breakthroughs in complex LLM reasoning within verifiable domains, such as mathematics and programming. Recent efforts have sought to extend this paradigm to open-ended tasks by employing LLMs-as-a-Judge to provide sequence-level rewards for policy optimization. However, these rewards are inherently sparse, failing to provide the fine-grained supervision necessary for generating complex, long-form trajectories. Furthermore, current work treats the Judge as a black-box oracle, discarding the rich intermediate feedback signals encoded in it. To address these limitations, we introduce Grad2Reward, a novel framework that extracts dense process rewards directly from the Judge's model inference process via a single backward pass. By leveraging gradient-based attribution, Grad2Reward enables precise token-level credit assignment, substantially enhancing training efficiency and reasoning quality. Additionally, Grad2Reward introduces a self-judging mechanism, allowing the policy to improve through its own evaluative signals without training specialized reward models or reliance on superior external Judges. The experiments demonstrate that policies optimized with Grad2Reward achieve outstanding performance across diverse open-ended tasks, affirming its effectiveness and broad generalizability.

Problem

• sparse rewards
• open-ended reasoning
• LLM-as-a-Judge
• fine-grained supervision
• intermediate feedback

Innovation

• dense rewards
• gradient-based attribution
• self-judging
• LLM-as-a-Judge
• token-level credit assignment