Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images

2025-06-08

AI Summary

Multimodal large language models (MLLMs) suffer from pervasive behavioral hallucinations—erroneous inferences about dynamic actions—in temporal image understanding. This work is the first to systematically identify their root causes: prior-driven bias and a snowball effect. Method: We propose SHE, a lightweight two-stage framework that integrates adaptive temporal window alignment detection with joint embedding-space orthogonal projection to decouple behavioral representations and suppress hallucinations. Contribution/Results: We introduce BEACH, the first dedicated evaluation metric for behavioral hallucinations. Extensive experiments on standard benchmarks demonstrate that SHE reduces BEACH scores by over 10% while preserving visual-language description accuracy—confirming hallucination suppression without compromising fidelity. This work establishes a new paradigm and scalable technical pathway for trustworthy temporal multimodal reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

While multimodal large language models excel at various tasks, they still suffer from hallucinations, which limit their reliability and scalability for broader domain applications. To address this issue, recent research mainly focuses on objective hallucination. However, for sequential images, besides objective hallucination, there is also behavioral hallucination, which is less studied. This work aims to fill in the gap. We first reveal that behavioral hallucinations mainly arise from two key factors: prior-driven bias and the snowball effect. Based on these observations, we introduce SHE (Sequence Hallucination Eradication), a lightweight, two-stage framework that (1) detects hallucinations via visual-textual alignment check using our proposed adaptive temporal window and (2) mitigates them via orthogonal projection onto the joint embedding space. We also propose a new metric (BEACH) to quantify behavioral hallucination severity. Empirical results on standard benchmarks demonstrate that SHE reduces behavioral hallucination by over 10% on BEACH while maintaining descriptive accuracy.

Problem

• Addressing behavioral hallucination in multimodal language models for sequential images
• Identifying prior-driven bias and snowball effect as key causes
• Proposing SHE framework to detect and mitigate behavioral hallucinations

Innovation

• Lightweight two-stage framework SHE
• Adaptive temporal window alignment
• Orthogonal projection mitigation