Transcoders Trace Visual Grounding and Hallucinations in Vision-Language Models

2026-05-21

AI Summary

Current vision-language models suffer from a lack of interpretability in visual-to-text mapping, obscuring the dynamic mechanisms underlying cross-modal interactions. This work introduces, for the first time, a functionally grounded Transcoder framework into this domain, constructing an interpretable computational pathway from image patches to generated tokens by sparsely approximating MLP sublayers. Integrating patch ablation, counterfactual analysis, and graph-structured circuit tracing, the proposed method not only stably and accurately attributes the visual grounding process but also enables mechanistic tracing of cross-modal hallucinations. Specifically, a graph-based feature classifier effectively predicts generation hallucinations with an AUC of 0.68, demonstrating the framework’s capacity to uncover the internal causes of such phenomena.

Abstract

Generative Vision-Language Models (VLMs) perform well on multimodal reasoning, but how visual inputs are transformed to text remains poorly understood. Existing interpretability work on VLMs uses Sparse Autoencoders (SAEs), which decompose static residual representations and miss the functional updates that drive cross-modal interaction. We adopt a function-centric framework based on Transcoders, sparse approximations of MLP sublayers that act as a causal proxy for layer-wise computation. Applied to Gemma 3-4B-IT, the framework decomposes the model into interpretable computational pathways linking image patches to directions in token generation. Transcoder attributions produce stronger and more stable effects on visually grounded tokens under patch ablation than SAE attributions, and align better with semantically relevant image regions. A False Visual Grounding counterfactual analysis confirms that the recovered pathways are specific to vision-language interaction.Finally, we perform a structural analysis of hallucinated generations, by extracting graph-based indicators from circuit traces produced by the transcoders. A logistic classifier over these mechanistic graph features predicts hallucinations at AUC $0.68$. These results show that function-centric circuit decomposition yields interpretable and predictive accounts of multimodal computation in VLMs.

Problem

• Vision-Language Models
• Visual Grounding
• Hallucinations
• Interpretability
• Multimodal Computation

Innovation

• Transcoders
• Vision-Language Models
• Interpretability
• Visual Grounding
• Hallucination Detection