DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research

2025-11-24

AI Summary

Existing open deep research models predominantly rely on reinforcement learning with verifiable rewards (RLVR), which is effective only for short-answer tasks and fails to scale to open-ended, long-text research requiring multi-step reasoning, evidence attribution, and long-range dependency modeling. Method: We propose Reinforcement Learning with Evolvable Rating criteria (RLER), a framework wherein evaluation criteria co-evolve with the policy model, enabling fine-grained, intra-policy feedback and supporting end-to-end training of open-source long-form deep research models. Integrated with the MCP agent architecture, our model supports multi-step reasoning and evidence provenance. Contribution/Results: We release DR-Tulu-8B, which substantially outperforms prior open models across four long-research benchmarks in scientific, medical, and general domains—matching or exceeding closed-source systems while using fewer parameters and lower inference cost. All data, models, and code are fully open-sourced.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Deep research models perform multi-step research to produce long-form, well-attributed answers. However, most open deep research models are trained on easily verifiable short-form QA tasks via reinforcement learning with verifiable rewards (RLVR), which does not extend to realistic long-form tasks. We address this with Reinforcement Learning with Evolving Rubrics (RLER), in which we construct and maintain rubrics that co-evolve with the policy model during training; this allows the rubrics to incorporate information that the model has newly explored and to provide discriminative, on-policy feedback. Using RLER, we develop Deep Research Tulu (DR Tulu-8B), the first open model that is directly trained for open-ended, long-form deep research. Across four long-form deep research benchmarks in science, healthcare and general domains, DR Tulu substantially outperforms existing open deep research models, and matches or exceeds proprietary deep research systems, while being significantly smaller and cheaper per query. To facilitate future research, we release all data, models, and code, including our new MCP-based agent infrastructure for deep research systems.

Problem

• Addressing limitations of RLVR training for long-form deep research tasks
• Developing evolving rubrics that co-adapt with policy models during training
• Creating first open model specialized for open-ended long-form research

Innovation

• Evolving rubrics co-train with policy model
• On-policy feedback using newly explored information
• MCP-based agent infrastructure for deep research