Mind the Heads: Topological Representation Alignment for Multimodal LLMs

2026-06-22

AI Summary

Existing multimodal large language models typically perform coarse-grained representation alignment only at fixed linguistic layers, overlooking the fine-grained structure within individual Transformer attention heads, which limits cross-modal alignment efficacy. This work proposes HeRA, the first method to achieve cross-modal topological alignment at the level of individual attention heads. Grounded in the Bregman representation hypothesis, HeRA enhances alignment quality by preserving local neighborhood relationships across modalities. It introduces a differentiable Mutual K-Nearest Neighbor (MKNN) contrastive objective that dynamically identifies and optimizes critical attention heads, revealing that the least-aligned heads yield the greatest performance gains. Experiments demonstrate that HeRA significantly improves visual-centric task performance across multiple state-of-the-art multimodal large language models and 18 benchmarks, while effectively mitigating visual hallucination and overreliance on linguistic priors.

Abstract

Representation alignment has emerged as an effective approach to improve Multimodal Large Language Models (MLLMs) by regularizing their internal representations toward those of an external vision encoder. However, existing methods typically align a fixed layer of the language backbone, overlooking the fine-grained structure of Transformer models. In this work, we propose Head-Wise Representation Alignment (HeRA), a method that enforces cross-modal alignment at the level of individual attention heads. Our approach is grounded in the Platonic Representation Hypothesis, focusing on preserving the topological structure of representations (i.e., their local neighborhood relationships) across modalities. Following the Mutual K-Nearest Neighbor (MKNN) alignment metric, we introduce a contrastive objective that acts as a differentiable proxy for matching local structures. HeRA applies this objective during multimodal training to specific attention heads in the LLM, selected by their alignment score according to the MKNN metric. Counterintuitively, we find that aligning the least aligned heads yields the largest gains. Extensive evaluations across multiple MLLMs and 18 benchmarks demonstrate that HeRA consistently improves performance on challenging vision-centric tasks and serves as an effective regularizer against visual hallucinations by naturally curbing the over-reliance on linguistic priors. Our code is publicly released.

Problem

• representation alignment
• multimodal LLMs
• attention heads
• topological structure
• visual hallucinations

Innovation

• Head-Wise Alignment
• Topological Representation
• Multimodal LLMs
• Mutual K-Nearest Neighbor
• Attention Heads