NLL-Guided Full-Attention Layer Selection for Training-Free Sliding-Window Adaptation

2026-06-26

AI Summary

This work addresses the challenge of efficiently selecting which layers to retain full attention in hybrid attention models, balancing long-context reasoning efficiency with downstream task accuracy. The authors propose a training-free layer selection method that performs a one-time, rapid calibration (approximately 15 minutes) to estimate each layer’s importance by measuring the degradation in negative log-likelihood (NLL) of answer tokens when replacing full attention with sliding window attention. This approach uniquely leverages NLL degradation as a direct proxy for downstream performance, eschewing fixed patterns or heuristic strategies. Experiments demonstrate that on Qwen3-4B and LongMemEval, using only one-quarter of the layers with full attention achieves 64.6% accuracy—matching the performance of a periodic baseline employing half full-attention layers and significantly outperforming other sparse attention methods.

Abstract

Hybrid attention models that mix full and sliding-window attention across layers offer a promising approach to efficient long-context inference, but the critical question of \emph{which layers} should retain full attention remains unsolved. Existing methods use either fixed periodic patterns or attention-based heuristics that may not capture what matters for downstream accuracy. We propose NLL-guided layer selection, a training-free method that directly measures each layer's importance by computing the negative log-likelihood degradation on answer tokens when that layer uses sliding-window instead of full attention. On LongMemEval with Qwen3-4B, our method achieves 64.6\% accuracy using only 1/4 full-attention layers, matching the 1/2-FA periodic baseline (65.0\%) while halving the computational budget. NLL-guided selection outperforms the SWAA-reported periodic 1/4-FA baseline by 10.4 percentage points and a matched LightTransfer-style baseline by 26.4 percentage points. De-confounding analysis shows the signal is consistent with long-range attention needs rather than generic layer sensitivity. The method requires only $\sim$15 minutes of one-time calibration, advancing the efficiency-accuracy Pareto frontier for long-context LLM deployment.

Problem

• layer selection
• full attention
• sliding-window attention
• long-context inference
• attention mechanism

Innovation

• NLL-guided layer selection
• sliding-window attention
• training-free adaptation
• long-context inference
• attention efficiency