Utilizing and Calibrating Hindsight Process Rewards via Reinforcement with Mutual Information Self-Evaluation

2026-04-13

AI Summary

This work addresses the challenge of learning in sparse-reward environments for large language model (LLM) reinforcement learning agents by proposing MISE, a method that generates dense rewards through post-hoc generative self-evaluation and calibrates them using environmental feedback to enable autonomous learning. It establishes the first theoretical foundation for the generative self-rewarding paradigm, proving its equivalence to a joint optimization objective involving mutual information and KL divergence, and leverages this insight to design an effective reward calibration mechanism. Experimental results demonstrate that MISE enables a 7B-parameter open-source LLM to achieve performance approaching that of GPT-4o without any expert supervision, significantly outperforming strong existing baselines.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions

Abstract

To overcome the sparse reward challenge in reinforcement learning (RL) for agents based on large language models (LLMs), we propose Mutual Information Self-Evaluation (MISE), an RL paradigm that utilizes hindsight generative self-evaluation as dense reward signals while simultaneously calibrating them against the environmental feedbacks. Empirically, MISE enables an agent to learn autonomously from dense internal rewards supplementing sparse extrinsic signals. Theoretically, our work provides the first formal foundation for the paradigm of generative self-rewarding. We prove that utilizing hindsight self-evaluation rewards is equivalent to minimizing an objective that combines mutual information with a KL divergence term between the policy and a proxy reward policy. This theoretical insight then informs and justifies our calibration step, which actively aligns these rewards with the optimal policy. Extensive experiments show that MISE outperforms strong baselines, enabling open-source LLMs about 7B parameters to achieve performance comparable to GPT-4o on validation without expert supervision.

Problem

• sparse reward
• reinforcement learning
• large language models
• hindsight self-evaluation
• dense reward signals

Innovation

• Mutual Information Self-Evaluation
• hindsight self-rewarding
• dense reward calibration
• generative self-rewarding
• reinforcement learning with LLMs