Disentangling Hallucinations: Orthogonal Semantic Projection for Robust Interpretability

2026-06-08

AI Summary

This work addresses the failure of explainability methods for vision-language models due to semantic hallucination—where irrelevant image regions are highlighted even under incorrect textual prompts. We present the first formal analysis of this issue through a theoretical framework termed Linear Semantic Attribution (LSA), which reveals that the root cause lies in linear semantic leakage within the embedding space. Building on this insight, we propose Orthogonal Semantic Projection (OSP), a geometric intervention mechanism that orthogonalizes query vectors with respect to confounding concepts, thereby disentangling unique semantics from shared features. Both theoretical analysis and extensive experiments demonstrate that OSP effectively suppresses semantic hallucination, significantly enhancing the faithfulness, robustness, and trustworthiness of explanations across diverse vision-language models and existing XAI methods.

Abstract

As Vision-Language Models are increasingly deployed in safety-critical applications, the trustworthiness of their explanations becomes crucial. Explainable AI (XAI) methods for Vision-Language Models often suffer from semantic hallucination, where attribution maps highlight prominent image regions even when prompted with incorrect text descriptions (e.g., highlighting a dog when prompted ``cat''). Although this problem is widespread, a formal mathematical analysis of XAI methods and CLIP embeddings is largely missing in the literature. We demonstrate that this phenomenon is not specific to a single architecture but is a fundamental consequence of Linear Semantic Leakage in high-dimensional embedding spaces. We propose a unified theoretical framework, Linear Semantic Attribution (LSA), which generalizes across discriminative methods. We introduce OSP, a geometric intervention that utilizes the residual property of OMP to disentangle unique semantic signals from shared concepts. We prove theoretically and demonstrate empirically that OSP minimizes hallucination by orthogonalizing the query vector against distractor concepts, rendering the attribution model blind to shared features while preserving fidelity for correct prompts. Our code is available at: https://github.com/emirhanbilgic/Orthogonal-Semantic-Projection

Problem

• semantic hallucination
• Vision-Language Models
• Explainable AI
• attribution maps
• CLIP embeddings

Innovation

• Orthogonal Semantic Projection
• Semantic Hallucination
• Linear Semantic Leakage
• Vision-Language Models
• Explainable AI