STAMP: Provenance-Guided Credit Assignment for Deep Search Agents

2026-07-13

AI Summary

This work addresses the misalignment in credit assignment between actions and rewards in deep search agents, where retrieving a document that supports reasoning receives no targeted reward. To resolve this, the authors propose a provenance-based fine-grained credit assignment mechanism. This approach employs a reference validator to assess whether a retrieved document supports entities or relations in the evidence graph, and attributes credit to the action that first retrieved the document via first-exposure attribution. The credit signal is further modulated by sign-preserving advantage injection. Without altering trajectory-level rewards, the method remains compatible with various base reward schemes and consistently outperforms the GRPO baseline by 2.0, 5.5, and 3.0 points on BrowseComp, BrowseComp-ZH, and xbench-DS, respectively.

Abstract

Reinforcement learning for deep-search agents has largely focused on trajectory-level scoring -- outcome correctness, citation-aware rewards, and evidence coverage. Yet the actions that expose supporting documents receive no targeted credit, a gap we call the reward-credit mismatch. We propose STAMP, in which a reference-based verifier judges whether each cited document supports an entity or relation in a training-time evidence graph, and first-exposure attribution traces each supported citation back to the action that first surfaced it. This step credit is injected through sign-preserving advantage modulation, which redistributes advantage across steps without changing the trajectory-level reward or the relative ranking of trajectories within each group. On BrowseComp, BrowseComp-ZH, and xbench-DS, STAMP improves the GRPO baseline by +2.0/+5.5/+3.0 points under matched SFT initialization, training data, and search tools, and composes with both outcome-only and citation-rubric base rewards. Component ablations confirm that the provenance-based credit signal and the sign-preserving advantage modulation each contribute to the gains.

Problem

• reward-credit mismatch
• credit assignment
• deep search agents
• reinforcement learning
• provenance

Innovation

• provenance-guided credit assignment
• sign-preserving advantage modulation
• first-exposure attribution
• deep search agents
• reinforcement learning