Towards Personalized Deep Research: Benchmarks and Evaluations

2025-09-29

AI Summary

Existing deep research agents (DRAs) are predominantly evaluated on closed-domain benchmarks, lacking systematic assessment for open-domain and personalized scenarios. To address this gap, we introduce the first personalized deep research benchmark—Personalized Deep Research Bench (PDRB)—comprising 250 realistic queries derived from 25 authentic user profiles across 10 domains and 50 tasks. We further propose the Personalized Query Rating (PQR) framework, enabling the first joint quantification of personalization alignment, content quality, and factual reliability. Methodologically, PQR integrates structured persona modeling with dynamic contextual simulation, supported by multi-dimensional human evaluation and automated verification. Experiments reveal substantial limitations in current DRAs’ capabilities for personalized understanding and deep reasoning. This work establishes the first reproducible, extensible benchmark for open-domain personalized deep research, providing both a rigorous evaluation standard and actionable optimization pathways for next-generation AI research assistants.

Technology Category

• Knowledge Representation and Reasoning: Qualitative Reasoning
• Search and Optimization: Metareasoning and Metaheuristics
• Data Mining & Knowledge Management: Intelligent Query Processing

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Deep Research Agents (DRAs) can autonomously conduct complex investigations and generate comprehensive reports, demonstrating strong real-world potential. However, existing evaluations mostly rely on close-ended benchmarks, while open-ended deep research benchmarks remain scarce and typically neglect personalized scenarios. To bridge this gap, we introduce Personalized Deep Research Bench, the first benchmark for evaluating personalization in DRAs. It pairs 50 diverse research tasks across 10 domains with 25 authentic user profiles that combine structured persona attributes with dynamic real-world contexts, yielding 250 realistic user-task queries. To assess system performance, we propose the PQR Evaluation Framework, which jointly measures (P) Personalization Alignment, (Q) Content Quality, and (R) Factual Reliability. Our experiments on a range of systems highlight current capabilities and limitations in handling personalized deep research. This work establishes a rigorous foundation for developing and evaluating the next generation of truly personalized AI research assistants.

Problem

• Evaluating personalization in autonomous deep research agents
• Addressing scarcity of open-ended personalized research benchmarks
• Assessing alignment, quality and reliability in personalized investigations

Innovation

• Introduced Personalized Deep Research Bench benchmark for DRAs
• Proposed PQR Framework evaluating Personalization Quality Reliability
• Combined user profiles with research tasks across domains