DEEPRUBRIC: Evidence-Tree Rubric Supervision for Efficient Reinforcement Learning of Deep Research Agents

2026-06-15

AI Summary

This work addresses the limitations of existing reward-based reinforcement learning approaches, wherein automatically generated rubrics often fail to accurately capture the information needs of queries, leading to unreliable reward signals and inefficient training. To overcome this, the authors propose DeepRubric, a novel framework that inverts the conventional pipeline: it first recursively constructs verifiable evidence trees to precisely define evaluation objectives, then synthesizes query–rubric pairs that are strictly aligned with these objectives, thereby providing high-quality supervision signals for reinforcement learning. Integrating this approach with a rubric-guided GRPO algorithm to train an 8B-parameter model, DeepRubric achieves state-of-the-art performance among open-source deep research models on three benchmarks while using only approximately one-thirteenth of the RL GPU-hours, substantially improving both data efficiency and training stability.

Abstract

Deep research agents synthesize long-form reports by searching and reasoning over retrieved evidence. Reinforcement learning with rubric-based rewards improves these agents by optimizing them against checkable criteria that translate report quality into reward signals, but its efficiency depends on whether those criteria reliably capture the task scope and evidence needs. Most existing studies ask an LLM to generate rubrics for a given query, but when the model fails to infer the underlying information needs, the generated rubrics may be incomplete and reduce RL efficiency. To obtain more reliable query--rubric supervision, we introduce DeepRubric, a data construction framework that reverses this process: instead of inferring evaluation criteria for a given query, it first determines what an evidence-backed report should be evaluated on and then synthesizes aligned query--rubric pairs from those evaluation targets. Starting from a sampled seed topic, DeepRubric builds an evidence tree by recursively expanding evidence-backed sub-questions, whose leaves serve as atomic and verifiable evaluation targets. It then uses the evidence tree to synthesize the training query and rubrics, ensuring that the reward evaluates exactly the information requested by the query. Using DeepRubric, we construct 9K query--rubric supervision examples and train DeepRubric-8B with rubric-based GRPO, achieving comparable performance to prior open state-of-the-art deep research models across three benchmarks with roughly 13x fewer RL GPU-hours.

Problem

• reinforcement learning
• rubric-based rewards
• deep research agents
• evidence reasoning
• query-rubric supervision

Innovation

• Evidence-Tree
• Rubric Supervision
• Reinforcement Learning Efficiency
• Query-Rubric Alignment
• Deep Research Agents