HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Heads

2026-04-18

AI Summary

This work addresses the loss of fine-grained discriminative capability in large language models during reranking due to attention weight homogenization. It introduces, for the first time, listwise preference alignment into a continuous attention space, proposing a decoding-free, efficient reranking framework. Discriminative power in homogeneous regions is enhanced through entropy-regularized attention head selection, hard adjacent-level pairwise preference construction, and distributional regularization, while depth truncation enables constant-time forward passes for inference. Evaluated on 14 benchmarks with only 211 training queries, the Qwen3 series (0.6B–4B) consistently outperforms existing baselines; notably, the 4B model ranks relevant intermediate documents within the top quartile 57.4% of the time—substantially higher than the 14.2% rate for irrelevant documents.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web search models and ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Decoding-free reranking methods that read relevance signals directly from LLM attention weights offer significant latency advantages over autoregressive approaches, yet suffer from attention score homogenization: middle-context documents receive near-identical scores, destroying the fine-grained distinctions required for ranking. We propose HeadRank, a framework that lifts preference optimization from discrete token space into the continuous attention domain through entropy-regularized head selection, hard adjacent-level preference pairs, and a distribution regularizer that jointly sharpen discriminability in the homogenized middle zone. Depth truncation at the deepest selected layer further reduces inference to $\mathcal{O}(1)$ forward passes. Across 14 benchmarks on three Qwen3 scales (0.6B--4B) using only 211 training queries, HeadRank consistently outperforms generative and decoding-free baselines with 100\% formatting success. At 4B, 57.4\% of relevant middle-zone documents reach the top quartile versus 14.2\% for irrelevant ones -- a 43-percentage-point selectivity gap that demonstrates the effectiveness of attention-space preference alignment for listwise reranking.

Problem

• attention homogenization
• decoding-free reranking
• passage reranking
• preference alignment
• LLM attention weights

Innovation

• decoding-free reranking
• preference-aligned attention
• attention homogenization
• entropy-regularized head selection
• listwise reranking