Phantom References: Hallucinated Citations That Survive Peer Review at Top-Tier Conferences

2026-07-01

AI Summary

This work addresses the growing threat of hallucinated citations—fabricated references generated by large language models—that have infiltrated top-tier academic venues such as ICLR, ICML, NeurIPS, and USENIX Security, undermining scholarly credibility. To combat this, the authors introduce RefChecker, the first scalable citation verification pipeline tailored for large-scale analysis of conference papers. RefChecker integrates multi-source academic database matching with web-based re-verification to efficiently assess citation authenticity under conservative criteria. The study presents the first systematic quantification of citation hallucinations under a rigorous definition, revealing that approximately 5% of NeurIPS and USENIX Security papers—including some award-winning works—contain at least two hallucinated references. Notably, the approach achieves audit costs as low as $0.04 per paper, demonstrating that automated, low-cost, and reproducible large-scale citation auditing is both feasible and practical.

Abstract

Large language models can generate polished scientific text that includes unsupported claims, allowing hallucinations to enter the archival record. Assessing this risk via technical statements is difficult and often requires expert judgment, but citations provide a more auditable surface: a reference either resolves to a real scholarly work with compatible authorship, or it does not. We measure citation hallucination in peer-reviewed proceedings using a conservative definition limited to identity-level failures: non-existent works and substantial author-list mismatches. We explicitly exclude ordinary bibliographic drift (e.g., venue/year differences, publication-status updates, minor name variants). To audit citations at scale, we build RefChecker, a verification pipeline that resolves bibliography entries against multiple bibliographic sources and escalates unresolved cases to web-search re-verification. We apply RefChecker to accepted camera-ready papers from ICLR, ICML, NeurIPS, and USENIX Security. Hallucinated citations have entered the archival record. While reference-level rates are usually below 1%, proceedings are large enough that paper-level failures are visible: in 2025, roughly one in twenty NeurIPS and USENIX Security papers contains at least two likely hallucinated academic-paper-like references under our strict definition. We also observe post-ChatGPT increases in several venues, including a tail of papers with 5+ failures in a single bibliography, and likely hallucinated citations even among award-winning papers. These results suggest peer review alone does not reliably enforce citation integrity, yet auditing is tractable (about 0.04$ per paper in one venue-scale scan). We open-source RefChecker for routine, reproducible citation verification before publication (https://github.com/markrussinovich/refchecker).

Problem

• citation hallucination
• peer review
• academic integrity
• large language models
• bibliographic verification

Innovation

• citation hallucination
• reference verification
• RefChecker
• peer review integrity
• large language models