Taming the Tri-Space Tension: ARC-Guided Hallucination Modeling and Control for Text-to-Image Generation

2025-07-07

AI Summary

Text-to-image (T2I) diffusion models suffer persistently from semantic hallucinations—deviations from the prompt’s underlying semantics, structural constraints, or commonsense knowledge. To address this, we propose a cognition-inspired “three-axis hallucination space,” modeling hallucination as alignment tension trajectories in the latent space along semantic, structural, and knowledge dimensions. We introduce the Alignment Risk Code (ARC), the first dynamic metric quantifying such multi-dimensional tension, and design a lightweight Tension Modulator that enables axis-specific intervention during sampling—without model retraining. Our method is architecture-agnostic and integrates seamlessly with mainstream diffusion pipelines. Evaluated across multiple T2I benchmarks, it reduces hallucination rates by an average of 32.7% while preserving image fidelity and diversity. The approach demonstrates strong effectiveness, interpretability, and deployment efficiency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Despite remarkable progress in image quality and prompt fidelity, text-to-image (T2I) diffusion models continue to exhibit persistent "hallucinations", where generated content subtly or significantly diverges from the intended prompt semantics. While often regarded as unpredictable artifacts, we argue that these failures reflect deeper, structured misalignments within the generative process. In this work, we propose a cognitively inspired perspective that reinterprets hallucinations as trajectory drift within a latent alignment space. Empirical observations reveal that generation unfolds within a multiaxial cognitive tension field, where the model must continuously negotiate competing demands across three key critical axes: semantic coherence, structural alignment, and knowledge grounding. We then formalize this three-axis space as the extbf{Hallucination Tri-Space} and introduce the Alignment Risk Code (ARC): a dynamic vector representation that quantifies real-time alignment tension during generation. The magnitude of ARC captures overall misalignment, its direction identifies the dominant failure axis, and its imbalance reflects tension asymmetry. Based on this formulation, we develop the TensionModulator (TM-ARC): a lightweight controller that operates entirely in latent space. TM-ARC monitors ARC signals and applies targeted, axis-specific interventions during the sampling process. Extensive experiments on standard T2I benchmarks demonstrate that our approach significantly reduces hallucination without compromising image quality or diversity. This framework offers a unified and interpretable approach for understanding and mitigating generative failures in diffusion-based T2I systems.

Problem

• Addressing persistent hallucinations in text-to-image generation models
• Modeling misalignments across semantic, structural, and knowledge axes
• Controlling latent space tension to reduce generative failures

Innovation

• ARC quantifies alignment tension in real-time
• TM-ARC applies targeted axis-specific interventions
• Hallucination Tri-Space models three-axis cognitive tension