Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation

2026-07-01

AI Summary

Existing knowledge distillation methods, which rely solely on output token distributions, struggle to effectively compress multimodal large language models (MLLMs) and underutilize visual attention mechanisms. This work reveals for the first time that visual attention during the response phase is highly correlated with downstream task performance. To address this, we propose a token-level adaptive distillation strategy that aligns the visual attention distributions of teacher and student models via KL divergence and dynamically weights the distillation strength for each token based on its attention entropy, enabling fine-grained attention transfer. By moving beyond the limitations of uniform distillation objectives, our approach significantly outperforms existing methods across multiple multimodal benchmarks and substantially enhances the performance of lightweight MLLMs.

Abstract

While knowledge distillation (KD) is widely adopted for training lightweight models by leveraging supervision from larger teacher models, relying solely on output token distributions has proven insufficient for compressing Multimodal Large Language Models (MLLMs). Since output tokens are a byproduct of the model attending to visual inputs, prior works have explored explicitly distilling attention to provide a direct supervisory signal. While promising, the precise utility of which attention signals to distill remains under-explored. In this work, we challenge the conventional reliance on prompt-to-vision attention by revealing that downstream performance correlates strongly with response-to-vision attention similarity to the teacher, but negligibly with that of prompt-conditioned attention. Furthermore, we observe that attention distributions exhibit significant variance across individual tokens, indicating that a uniform distillation objective is suboptimal. To this end, we introduce Token-level Response-visual Attention Guidance (TRAG), a distillation objective that 1) shifts the focus to response-to-vision signals and 2) employs token-specific objectives by adaptively weighting the Kullback-Leibler divergence based on attention entropy, effectively guiding the student to mirror the teacher's precise visual focus. Extensive experimental results on multiple benchmarks demonstrate that TRAG significantly outperforms prior distillation baselines.

Problem

• knowledge distillation
• multimodal LLMs
• attention guidance
• response-to-vision attention
• token-level distillation

Innovation

• knowledge distillation
• multimodal LLMs
• attention guidance
• token-level distillation
• response-to-vision attention