🤖 AI Summary
This study addresses the prevalence of unreliable citations in in-depth research reports generated by large language models (LLMs), which often suffer from broken links, irrelevant content, or factual inaccuracies. The authors propose the first end-to-end verifiable evaluation framework specifically designed for LLM-generated citations: it extracts inline references from Markdown reports via abstract syntax tree (AST) parsing, retrieves source content using retrieval-augmented generation (RAG), and conducts a closed-loop assessment across three dimensions—link validity, content relevance, and factual accuracy. The framework reveals a significant disconnect between superficial citation quality and factual reliability: state-of-the-art models achieve over 94% link validity and more than 80% relevance, yet their factual accuracy ranges only from 39% to 77%. Moreover, deeper research synthesis correlates with an average 42% decline in factual accuracy. The work also releases a large-scale citation verification pipeline and human-calibrated scoring criteria.
📝 Abstract
Large language models (LLMs) power deep research agents that synthesize information from hundreds of web sources into cited reports, yet these citations cannot be reliably verified. Current approaches either trust models to self-cite accurately, risking bias, or employ retrieval-augmented generation (RAG) that does not validate source accessibility, relevance, or factual consistency. We introduce the first source attribution evaluation framework that uses a reproducible AST parser to extract and evaluate inline citations from LLM-generated Markdown reports at scale. Unlike methods that verify claims in isolation, our framework closes the loop by retrieving the actual cited content, enabling human or model evaluators to judge each citation against its source. Citations are evaluated along three dimensions. (1) Link Works verifies URL accessibility, (2) Relevant Content measures topical alignment, and (3) Fact Check validates factual accuracy against source content. We benchmark 14 closed-source and open-source LLMs across three evaluation dimensions using rubric-based LLM-as-a-judge evaluators calibrated through human review. Our results reveal that even the strongest frontier models maintain link validity above 94% and relevance above 80%, yet achieve only 39-77% factual accuracy, while fewer than half of open-source models successfully generate cited reports in a one-shot setting. Ablation studies on research depth show that Fact Check accuracy drops by approximately 42% on average across two frontier models as tool calls scale from 2 to 150, demonstrating that more retrieval does not produce more accurate citations. These findings reveal a critical disconnect between surface-level citation quality and factual reliability, and our framework provides the evaluation infrastructure to assess the disconnect.