Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning

2026-01-11

arXiv.org

AI Summary

This work addresses the significant structural redundancy observed in deep layers of large language models, which has lacked a functional explanation. We uncover, for the first time, the “BOS sink” phenomenon: attention heads with high BOS sink scores in deeper layers merely act as repositories for superfluous attention weights, exhibiting functional redundancy. Building on this insight, we propose a sink-aware pruning strategy that leverages BOS sink scores to identify and remove redundant attention heads, replacing conventional approaches based on weight or activation magnitudes. Experiments on Gemma-3, Llama-3.1, and Qwen3 demonstrate that our method more accurately locates redundant components, preserving performance close to that of dense models even under aggressive pruning, while maintaining robustness across varying sequence lengths.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large Language Models (LLMs) are known to contain significant redundancy, yet a systematic explanation for why certain components, particularly in higher layers, are more redundant has remained elusive. In this work, we identify the BOS sink phenomenon as a key mechanism driving this layer-wise sensitivity. We show that attention heads with high BOS sink scores are strongly associated with functional redundancy: such heads, especially in deeper layers, contribute little to predictive performance and effectively serve as \emph{dumping grounds} for superfluous attention weights. This provides a concrete functional explanation for the structural redundancy reported in prior studies. Leveraging this insight, we introduce a simple pruning strategy that removes high-BOS sink heads. Experiments on Gemma-3, Llama-3.1, and Qwen3 demonstrate that this approach identifies redundant transformer components more reliably than weight- or activation-based criteria, while preserving performance close to dense baselines even under aggressive pruning. Moreover, we find that the behavior of sink heads remains stable across different sequence lengths. Overall, our results suggest that structural properties of attention offer a more intuitive and robust basis for model compression than magnitude-based methods.

Problem

• redundancy
• large language models
• attention heads
• BOS sink
• model compression

Innovation

• BOS sink
• attention redundancy
• structured pruning
• model compression
• transformer heads