Dense Process Supervision for Search Agents via Fact Utility Estimation

2026-09-01

AI Summary

本文针对搜索代理中强化学习的信用分配问题，提出了一种基于事实效用估计的密集过程监督方法，通过将推理过程建模为离散证据事实的累积，并使用贝叶斯估计来推断每个事实簇的后验效用，从而提供密集步骤级别的奖励指导训练。

Abstract

Reinforcement learning (RL) for search agents typically relies on outcome rewards. However, it often fails to achieve effective credit assignment, due to the unclear value of intermediate steps. It is hard to separate their contributions from the final result. In this paper, we propose a dense process supervision method based on fact utility estimation, which models the reasoning process as the accumulation of discrete evidence facts. We first extract structured facts from raw observations and organize them into an explicit fact store. To support credit assignment, we then cluster semantically equivalent facts and infer the posterior utility of each fact cluster using Bayesian estimation over group rollouts. Finally, we convert the estimated fact utilities into dense step-level rewards to guide RL training. Experiments on seven single-hop and multi-hop QA benchmarks show that our method consistently outperforms existing baselines. Ablation studies validate clear relative improvements on multi-hop QA compared to outcome reward-only training.

Problem

• Reinforcement learning
• credit assignment
• intermediate steps

Innovation

• Dense Process Supervision
• Fact Utility Estimation
• Credit Assignment
• Bayesian Estimation
• Reinforcement Learning