Semantic Head Specialization Guides Hybrid ViT Attention for Multimodal LLMs

2026-08-28

AI Summary

研究通过提出SHS-Index量化视觉Transformer注意力头的语义专业化，设计了Ariadne Attention以减少计算同时保持性能。

Abstract

Hybrid attention dominates frontier LLMs, yet Vision Transformers (ViTs) in multimodal LLMs lack a satisfactory hybrid design, with no consensus on why certain attention patterns work better. To fill this gap, we study ViT attention heads and find they differentiate into object- and background-specialist roles, a pattern most pronounced under full attention; we call this Semantic Head Specialization (SHS). We propose SHS-Index to quantify this specialization, show that it distinguishes full-attention from chunk-window ViTs, and find that it strongly tracks downstream benchmark performance. We then identify three structural factors that shape SHS---window interaction, token serialization, and local softmax allocation---and use them as design principles for hybrid attention. Guided by these factors, we design Ariadne Attention, a hybrid that matches full attention on 22 image and video tasks at 6.5x less attention compute. Our findings establish head specialization as a measurable property for diagnosing and designing principled hybrid ViT attention at the multimodal-LLM scale.

Problem

• Hybrid Attention
• Vision Transformers
• Multimodal LLMs
• Semantic Head Specialization

Innovation

• Semantic Head Specialization
• SHS-Index
• Ariadne Attention
• Hybrid Attention