Llama See, Llama Do: A Mechanistic Perspective on Contextual Entrainment and Distraction in LLMs

2025-05-14

AI Summary

This work identifies and systematically investigates “contextual entrainment” in large language models: a mechanistic phenomenon wherein the model uniformly elevates logits for any token previously appearing in the prompt—regardless of relevance—thereby inducing spurious context interference. The effect is modulated by semantic factors. We first propose and empirically validate the “entrainment heads” hypothesis, developing a suite of methods including differentiable attention-head masking for identification, counterfactual prompting for causal analysis, and logit-bias quantification; we further perform causal intervention via targeted attention-head ablation. Across multiple state-of-the-art models, our approach precisely localizes entrainment heads; post-intervention, context-induced interference diminishes significantly, and model outputs revert to baseline performance levels observed in interference-free settings. This work provides the first interpretable, empirically verifiable, mechanism-level framework for diagnosing and mitigating attentional distraction in LMs.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We observe a novel phenomenon, contextual entrainment, across a wide range of language models (LMs) and prompt settings, providing a new mechanistic perspective on how LMs become distracted by ``irrelevant'' contextual information in the input prompt. Specifically, LMs assign significantly higher logits (or probabilities) to any tokens that have previously appeared in the context prompt, even for random tokens. This suggests that contextual entrainment is a mechanistic phenomenon, occurring independently of the relevance or semantic relation of the tokens to the question or the rest of the sentence. We find statistically significant evidence that the magnitude of contextual entrainment is influenced by semantic factors. Counterfactual prompts have a greater effect compared to factual ones, suggesting that while contextual entrainment is a mechanistic phenomenon, it is modulated by semantic factors. We hypothesise that there is a circuit of attention heads -- the entrainment heads -- that corresponds to the contextual entrainment phenomenon. Using a novel entrainment head discovery method based on differentiable masking, we identify these heads across various settings. When we ``turn off'' these heads, i.e., set their outputs to zero, the effect of contextual entrainment is significantly attenuated, causing the model to generate output that capitulates to what it would produce if no distracting context were provided. Our discovery of contextual entrainment, along with our investigation into LM distraction via the entrainment heads, marks a key step towards the mechanistic analysis and mitigation of the distraction problem.

Problem

• Investigates contextual entrainment in LMs causing distraction by irrelevant input tokens.
• Identifies entrainment heads responsible for LM distraction via mechanistic analysis.
• Proposes method to mitigate distraction by modulating entrainment heads' influence.

Innovation

• Identifies contextual entrainment in language models
• Uses differentiable masking to discover entrainment heads
• Mitigates distraction by turning off entrainment heads