Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads

2026-07-01

AI Summary

Existing methods struggle to identify attention heads in large language models that perform non-literal retrieval—i.e., synthesizing answers based on semantic composition—because they focus solely on read locations while neglecting the write role of output-value (OV) circuits. This work proposes a write-aware Logit-Contribution Scoring (LOCOS) method that precisely localizes critical attention heads in a single forward pass by quantifying each head’s contribution to the logit along the direction of the target token embedding. Experiments on models such as Qwen3, Gemma-3, and OLMo-3.1 demonstrate that ablating LOCOS-identified heads causes a drastic drop in ROUGE-L scores (e.g., from 0.401 to 0.000 in Qwen3-8B), substantially outperforming baseline approaches, while preserving the model’s capacity for parameter memorization and arithmetic reasoning.

Abstract

In long-context use, large language models frequently synthesize answers from the meaning of a relevant context span rather than literally copy-pasting them. Identifying which attention heads perform this synthesis matters for interpreting long-context model behavior. Yet existing detectors miss these heads by construction: they reward heads whose attended token matches the generated token, a literal-copy criterion that captures where a head reads but not what it writes through its output-value (OV) circuit, the very mechanism that carries non-literal retrieval. We introduce Logit-Contribution Scoring (LOCOS), a write-aware detector that scores each head by the projection of its OV-circuit output onto the answer-token unembedding direction, contrasting needle and off-needle source positions in a single forward pass. Across three model families (Qwen3, Gemma-3, OLMo-3.1), mean-ablating the top LOCOS heads on the NoLiMa non-literal retrieval benchmark collapses ROUGE-L at lower head counts than prior attention-based detections; on Qwen3-8B, ablating 50 heads drives ROUGE-L from 0.401 to 0.000 while the strongest baseline still retains 0.292. The selected heads are retrieval-specific: parametric recall and arithmetic reasoning stay at baseline under the same ablation. On Qwen3-8B, the same ablation also drops MuSiQue from 0.55 to 0.08 and BABI-Long from 0.62 to 0.20, while a random-heads control stays within 0.05 of baseline.

Problem

• non-literal retrieval
• attention heads
• logit contribution
• OV circuit
• long-context

Innovation

• Logit-Contribution Scoring
• non-literal retrieval
• attention heads
• OV circuit
• write-aware detection