Who Checks the Citations? Benchmarking Legal Hallucination Detection

2026-06-19

AI Summary

This study addresses the growing concern of legal hallucinations—fabricated citations—in AI-generated legal documents, for which effective detection methods are urgently needed. The authors introduce the first taxonomy of legal citation hallucinations grounded in real court filings and release the first benchmark dataset specifically designed for detecting such errors, comprising 1,300 expert-annotated samples. Evaluating large language models, including the GPT series, under both agentic and non-agentic frameworks with multi-step reasoning and external retrieval, they find that GPT-5 achieves a recall of 82.8% and an F1 score of 60.5% in agentic mode. However, identifying subtle citation inaccuracies remains challenging, and reliance on proprietary legal databases limits the fairness and practical applicability of current approaches.

Abstract

Attorneys, judges, and pro se filers increasingly use AI to draft legal documents, yet these tools frequently fabricate citations. Despite predictions that newer models would hallucinate less or that court sanctions would deter negligent filers, we found over 1,000 filings containing fabricated citations -- with this number growing year-over-year. This study evaluates whether AI-based systems can mitigate these errors by automatically detecting hallucinations. We propose a taxonomy of legal citation hallucinations grounded in actual court filings and introduce a dataset of 1,300 brief excerpts containing injected errors. Benchmarking five models in agentic and non-agentic settings reveals that while the latest iterations perform better -- GPT-5 achieves 82.8% recall and a 60.5% F1 score in an agentic framework -- all models struggle with subtle error categories. Agentic verification remains resource-intensive, with GPT-5 averaging 16.9 steps per excerpt. Furthermore, restricted information access limits the efficacy of even the best agents. This gap creates policy concerns, as it disadvantages both AI systems and litigants who lack subscriptions to commercial legal databases. Together, our dataset, tools, and policy recommendations provide a foundation for building and auditing reliable legal citation checking tools.

Problem

• legal hallucination
• citation verification
• AI-generated legal documents
• fabricated citations
• hallucination detection

Innovation

• legal hallucination
• citation verification
• agentic AI
• benchmarking dataset
• GPT-5 evaluation