Innovation: An Almost Characterization of Hallucination

2026-05-26

AI Summary

This work investigates the fundamental causes of hallucination in large language models and their relationship with calibration, proposing “innovation”—defined as the model’s tendency to generate content outside the training distribution—as a core metric. Through probabilistic modeling and information-theoretic analysis, the study establishes a high-probability bidirectional implication between innovation and hallucination, demonstrating their near-equivalence. Building on this equivalence, the authors derive a novel lower bound on hallucination rate that jointly incorporates missing mass and innovation rate. This bound provides the first theoretical framework that almost entirely attributes hallucination to innovation, thereby significantly advancing the mechanistic understanding of hallucinatory behavior in language models.

Abstract

Hallucination is a central limitation of large language models (LLMs), and substantial effort has been devoted to understanding and mitigating it. Towards this, Kalai and Vempala (STOC 2024) introduced a probabilistic framework formalizing calibration and hallucination, and showed that, with high probability, calibrated LLMs hallucinate roughly at the rate of the "missing mass", a measure of how incomplete the training data is relative to its source. This raises two fundamental questions: (i) what property of a calibrated LLM makes hallucinations unavoidable? and (ii) can hallucinations be avoided by giving up calibration? We answer these questions by introducing a simpler property we call innovation that measures the tendency of a model to produce outputs outside the training data. We show that innovation is implied by the condition for hallucination identified by Kalai and Vempala, and, further, that it is an almost characterization of hallucination: hallucination implies innovation, and conversely, innovation implies hallucination with high probability. We also provide lower bounds on the hallucination rate based on the "innovation rate", and by relating innovation rate back to missing mass, we obtain new hallucination rate lower bounds based on missing mass that extend the results of Kalai and Vempala.

Problem

• hallucination
• large language models
• calibration
• innovation
• missing mass

Innovation

• innovation
• hallucination
• calibration
• missing mass
• large language models