MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents

2026-01-18

AI Summary

Existing benchmarks struggle to evaluate models’ ability to utilize evidence in end-to-end multimodal deep research. To address this gap, this work introduces a novel evaluation benchmark comprising 140 expert-designed tasks, each presenting an image–text pair and requiring the model to generate a research report grounded in explicit evidence and consistent across modalities. We propose the first comprehensive evaluation framework tailored for report-style multimodal deep research, featuring three fine-grained assessment modules: FLAE (report quality), TRACE (citation alignment), and MOSAIC (multimodal completeness), enabling diagnostic analysis. Experiments across 25 state-of-the-art models reveal systematic trade-offs among generation quality, citation fidelity, and multimodal grounding, highlighting multimodal completeness as a critical bottleneck.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Deep Research Agents (DRAs) generate citation-rich reports via multi-step search and synthesis, yet existing benchmarks mainly target text-only settings or short-form multimodal QA, missing end-to-end multimodal evidence use. We introduce MMDeepResearch-Bench (MMDR-Bench), a benchmark of 140 expert-crafted tasks across 21 domains, where each task provides an image-text bundle to evaluate multimodal understanding and citation-grounded report generation. Compared to prior setups, MMDR-Bench emphasizes report-style synthesis with explicit evidence use, where models must connect visual artifacts to sourced claims and maintain consistency across narrative, citations, and visual references. We further propose a unified, interpretable evaluation pipeline: Formula-LLM Adaptive Evaluation (FLAE) for report quality, Trustworthy Retrieval-Aligned Citation Evaluation (TRACE) for citation-grounded evidence alignment, and Multimodal Support-Aligned Integrity Check (MOSAIC) for text-visual integrity, each producing fine-grained signals that support error diagnosis beyond a single overall score. Experiments across 25 state-of-the-art models reveal systematic trade-offs between generation quality, citation discipline, and multimodal grounding, highlighting that strong prose alone does not guarantee faithful evidence use and that multimodal integrity remains a key bottleneck for deep research agents.

Problem

• Multimodal Deep Research
• Citation-grounded Report Generation
• Evidence Use
• Multimodal Integrity
• Benchmark

Innovation

• Multimodal Deep Research Agents
• MMDeepResearch-Bench
• Citation-grounded Report Generation
• Multimodal Evidence Alignment
• Interpretable Evaluation Pipeline