Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Art

2024-03-25

arXiv.org

AI Summary

Foundation models frequently exhibit “hallucinations” during autonomous decision-making, leading to high-risk misjudgments—yet no formal definition or systematic characterization of hallucination exists for decision tasks. Method: This work introduces the first task-specific definition of hallucination in decision-making and establishes a cross-task, scalable hallucination taxonomy; proposes a synergistic framework integrating uncertainty quantification with hallucination detection; and develops a joint detection–decision evaluation paradigm grounded in systematic survey analysis, probabilistic uncertainty modeling, and decision-chain interpretability. Results: We present the first comprehensive landscape of hallucination detection techniques tailored to decision contexts, contributing seven actionable implementation guidelines and identifying five critical research gaps—thereby advancing the safe deployment of trustworthy foundation models in high-stakes domains such as healthcare and transportation.

Technology Category

• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Machine Learning: Calibration & Uncertainty Quantification
• Cognitive Modeling & Cognitive Systems: Other Foundations of Cognitive Modeling & Systems

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Autonomous systems are soon to be ubiquitous, spanning manufacturing, agriculture, healthcare, entertainment, and other industries. Most of these systems are developed with modular sub-components for decision-making, planning, and control that may be hand-engineered or learning-based. While these approaches perform well under the situations they were specifically designed for, they can perform especially poorly in out-of-distribution scenarios that will undoubtedly arise at test-time. The rise of foundation models trained on multiple tasks with impressively large datasets has led researchers to believe that these models may provide"common sense"reasoning that existing planners are missing, bridging the gap between algorithm development and deployment. While researchers have shown promising results in deploying foundation models to decision-making tasks, these models are known to hallucinate and generate decisions that may sound reasonable, but are in fact poor. We argue there is a need to step back and simultaneously design systems that can quantify the certainty of a model's decision, and detect when it may be hallucinating. In this work, we discuss the current use cases of foundation models for decision-making tasks, provide a general definition for hallucinations with examples, discuss existing approaches to hallucination detection and mitigation with a focus on decision problems, present guidelines, and explore areas for further research in this exciting field.

Problem

• Detect hallucinations in foundation models for decision-making.
• Quantify certainty of model decisions in out-of-distribution scenarios.
• Review and improve hallucination detection and mitigation techniques.

Innovation

• Foundation models for decision-making
• Hallucination detection techniques
• Flexible definition of hallucinations