Hallucination Detector: A hybrid LLM and Semantic Scholar tool calling for detecting hallucination in scientific literature on AtomGPT.org

2026-07-07

AI Summary

Large language models (LLMs) frequently generate fabricated references in scholarly writing—such as fictitious authors or incorrect DOIs—a problem that has already infiltrated top-tier conferences and journals and often evades detection by conventional peer review. This work proposes the first automated detection tool that integrates LLM-based field extraction with structured querying of academic databases. Specifically, the method employs an LLM to parse citation fields and then leverages Semantic Scholar to perform semantic matching and similarity scoring based on title, author names, and publication venue, yielding a tiered credibility assessment (credible, partially supported, or likely fabricated). Evaluated on a manually annotated test set of papers accepted at NeurIPS 2025, the approach efficiently identifies the vast majority of hallucinated citations, offering a scalable technical safeguard for research integrity.

Abstract

Large language models are now commonly used as partners in scientific writing, and this shift has brought a subtler type of failure: made-up references. Fabricated authors, bogus DOIs, wrongly assigned identifiers, and citations that merge elements from multiple genuine articles are now being inserted into manuscripts at a volume that traditional peer review was never meant to handle. Recent audits reveal that such references have already slipped through the review process and made their way into the published literature, including leading journals and conferences. Automated verification that operates at the speed and scale of modern content production has therefore become a necessary safeguard rather than a convenience. This work presents and evaluates the AtomGPT reference checker (https://atomgpt.org/hallucination_detector), an open, web-accessible tool that verifies citations against the scholarly literature by combining large-language-model field extraction with structured retrieval from Semantic Scholar. For each reference, the tool extracts the bibliographic fields, retrieves the closest matching real papers, and scores the agreement across title, authorship, and venue to produce a graded judgment of whether a citation is trustworthy, partially supported, or likely fabricated. We benchmark the tool against an externally curated set of confirmed hallucinated citations from accepted NeurIPS 2025 papers and find that it reliably flags the great majority of them.

Problem

• hallucination
• scientific literature
• fake references
• large language models
• citation verification

Innovation

• hallucination detection
• large language models
• citation verification
• Semantic Scholar
• scientific integrity