Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty

2026-04-11

AI Summary

This work addresses the inefficiency and coarse evaluation inherent in existing generative reward models, which uniformly employ chain-of-thought reasoning regardless of input difficulty, leading to high computational costs and reliance on rudimentary voting mechanisms. To overcome these limitations, the authors propose E-GRM, a novel framework that leverages a model’s internal uncertainty as a universal signal to dynamically decide whether to activate chain-of-thought reasoning. E-GRM further introduces a lightweight discriminative scorer for fine-grained assessment of reasoning paths. The reward model is trained jointly with regression and ranking objectives, achieving high fidelity without requiring handcrafted features or task-specific customization. Experiments demonstrate that E-GRM consistently improves answer accuracy while substantially reducing computational overhead across multiple reasoning benchmarks, confirming its efficiency and broad applicability.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Reasoning under Uncertainty: Graphical Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Recent advancements in the Generative Reward Model (GRM) have demonstrated its potential to enhance the reasoning abilities of LLMs through Chain-of-Thought (CoT) prompting. Despite these gains, existing implementations of GRM suffer from two critical limitations. First, CoT prompting is applied indiscriminately to all inputs regardless of their inherent complexity. This introduces unnecessary computational costs for tasks amenable to fast, direct inference. Second, existing approaches primarily rely on voting-based mechanisms to evaluate CoT outputs, which often lack granularity and precision in assessing reasoning quality. In this paper, we propose E-GRM, an efficient generative reward modeling framework grounded in model-internal uncertainty. E-GRM leverages the convergence behavior of parallel model generations to estimate uncertainty and selectively trigger CoT reasoning only when needed, without relying on handcrafted features or task-dependent signals. To improve reward fidelity, we introduce a lightweight discriminative scorer trained with a hybrid regression--ranking objective to provide fine-grained evaluation of reasoning paths. Experiments on multiple reasoning benchmarks show that E-GRM substantially reduces inference cost while consistently improving answer accuracy, demonstrating that model-internal uncertainty is an effective and general signal for efficient reasoning-aware reward modeling.

Problem

• Generative Reward Model
• Chain-of-Thought
• model-internal uncertainty
• reasoning efficiency
• reward modeling

Innovation

• Generative Reward Modeling
• Model-Internal Uncertainty
• Chain-of-Thought Reasoning
• Efficient Inference
• Discriminative Scorer