Hallucination Inspector: A Fact-Checking Judge for API Migration

2026-04-22

AI Summary

This work addresses a critical yet previously underexplored issue in large language model (LLM)-assisted API migration: the generation of erroneous calling contexts containing fabricated symbols—such as nonexistent imports or constructors—termed “scaffolding hallucination.” Existing evaluation metrics struggle to detect such inaccuracies effectively. To tackle this problem, the paper formally defines scaffolding hallucination and introduces a lightweight, static analysis–based fact-checking approach. The method parses the abstract syntax tree of generated code to extract referenced symbols and validates them against a knowledge base constructed from official API documentation. Evaluated on Android API migration tasks, this technique significantly outperforms conventional evaluation metrics and probabilistic judgment methods, demonstrating high precision in identifying hallucinated code while substantially reducing false positives.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) are increasingly deployed in automated software engineering for tasks such as API migration. While LLMs are able to identify migration patterns, they often make mistakes and fail to produce correct glue code to invoke the new API in place of the old one. We call this issue Scaffolding Hallucination, a failure mode where models generate incorrect calling contexts by inventing Phantom Symbols -- such as imaginary imports, constructors, and constants -- that do not exist in the API specification. In this paper, we show that standard metrics cannot be relied upon to detect these instances of hallucination. We propose Hallucination Inspector, a static analysis tool to detect Scaffolding Hallucination in LLM-generated code. Our approach includes a lightweight evaluation framework that verifies symbols extracted from the abstract syntax tree against a knowledge base derived directly from software documentation for the API. A preliminary evaluation on Android API migrations demonstrates that our approach successfully identifies hallucinations and significantly reduces false positives compared to standard metrics and probabilistic judges

Problem

• Hallucination
• API Migration
• Large Language Models
• Scaffolding Hallucination
• Fact-Checking

Innovation

• Scaffolding Hallucination
• Hallucination Inspector
• API Migration
• Static Analysis
• Knowledge Base Verification