ReportLogic: Evaluating Logical Quality in Deep Research Reports

2026-01-27

arXiv.org

AI Summary

This work addresses a critical gap in existing large language model (LLM) evaluation frameworks, which often overlook the logical quality of in-depth research reports and fail to verify whether claims are supported by traceable and auditable evidence. To bridge this gap, the authors introduce ReportLogic, a novel benchmark featuring a three-tiered logical evaluation framework—encompassing macro-level coherence, explanatory reasoning, and structural organization—and construct a high-quality, human-annotated dataset. Leveraging this data, they train LogicJudge, an open-source discriminative model enabling scalable and reliable assessment of logical soundness. Experimental results demonstrate that general-purpose LLM evaluators are prone to biases from superficial features such as verbosity, whereas the proposed approach significantly enhances the fidelity and robustness of logical judgment, establishing a new paradigm for trustworthy evaluation of generated content.

Abstract

Users increasingly rely on Large Language Models (LLMs) for Deep Research, using them to synthesize diverse sources into structured reports that support understanding and action. In this context, the practical reliability of such reports hinges on logical quality: whether the report's claims and arguments are explicitly supported and can be trusted as a basis for downstream use, rather than merely appearing fluent or informative. However, current evaluation frameworks largely overlook this requirement. To bridge this gap, we introduce ReportLogic, a benchmark that quantifies report-level logical quality through a reader-centric lens of auditability. Specifically, ReportLogic adopts a hierarchical taxonomy that evaluates whether readers can (1) trace an on-topic report structure with a unified analytical arc (Macro-Logic), (2) understand the progression with necessary context (Expositional-Logic), and (3) verify conclusions via explicit claim--support (Structural-Logic). Based on this taxonomy, we construct a human-annotated rubric-guided dataset and train an open-source LogicJudge for scalable evaluation. We further evaluate judge robustness via adversarial attacks, showing that off-the-shelf LLM judges are frequently influenced by superficial cues (e.g., verbosity), and reasoning modes can mask broken support relations. Overall, our results provide actionable guidance for building more robust logic evaluators and improving the logical reliability of LLM-generated reports.

Problem

• logical quality
• Deep Research
• Large Language Models
• evaluation framework
• auditability

Innovation

• ReportLogic
• logical quality
• auditability
• LogicJudge
• hierarchical taxonomy