Vision Language Models Map Logos to Text via Semantic Entanglement in the Visual Projector

2025-10-14

AI Summary

This work identifies a novel “logo hallucination” phenomenon in vision-language models (VLMs), wherein models erroneously generate brand names for logo-only images lacking textual elements. We trace this failure to entanglement between glyph perception and symbolic semantics in the visual projector—not to actual text recognition. To address it, we propose a diagnostic framework based on embedding-layer analysis and targeted ablation, enabling the first precise localization of the critical projection subspace inducing hallucinations. Building on this, we design a subspace disentanglement mechanism and an OCR-guided decoding strategy. We systematically evaluate our approach across pure-symbol, hybrid, and text-containing logo datasets—including the challenging Hard-60 subset—under nine structured perturbations. Results show significant reduction in logo hallucination rates across mainstream VLMs while preserving OCR accuracy. Notably, circular logos are found to be most prone to such hallucinations, offering new insights and interpretable intervention pathways for multimodal robustness research.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Vision Language Models (VLMs) have achieved impressive progress in multimodal reasoning; yet, they remain vulnerable to hallucinations, where outputs are not grounded in visual evidence. In this paper, we investigate a previously overlooked setting: logo hallucination, where models generate brand names or textual content despite logos containing no visible words. Using curated splits of pure symbols, hybrids, and text-bearing logos, as well as the challenging Hard-60 subset, we systematically measure hallucination across leading VLMs. We further probe robustness through nine structured perturbations and show that hallucinations persist even under strong distortions, with occlusion exposing the sharpest weaknesses. Embedding-level analysis with open-weight LLaVA demonstrates that hallucination is tied to a small subset of projector dimensions, and targeted ablation substantially reduces errors while preserving OCR accuracy. Together, these findings reveal that VLMs often rely on symbolic priors rather than genuine glyph perception, particularly for iconic circular logos, and that projector subspaces play a decisive role in this failure mode. Our work contributes both a novel diagnostic lens and actionable mitigation insights, highlighting projector disentanglement and OCR-guided decoding as promising directions for building more trustworthy multimodal systems.

Problem

• VLMs hallucinate brand names from logos without visible text
• Logo hallucination persists under strong visual distortions and occlusions
• Semantic entanglement in visual projector dimensions causes false logo recognition

Innovation

• Targeted ablation reduces hallucination via projector dimensions
• Projector disentanglement mitigates logo hallucination in VLMs
• OCR-guided decoding enhances multimodal system trustworthiness