Mixing Mechanisms: How Language Models Retrieve Bound Entities In-Context

2025-10-07

AI Summary

This work investigates how language models bind and retrieve entities within context, particularly addressing robustness degradation when increased entity counts cause positional encoding to fail. Using attribution analysis, mechanistic probing, and causal modeling, we systematically evaluate nine mainstream models across ten binding tasks. We find that models dynamically coordinate three complementary mechanisms—positional cues, lexical similarity, and self-reflection—rather than relying on a single strategy. Building on this insight, we propose the first unified causal model that formally characterizes their interaction logic, achieving 95% token-level prediction consistency. The model demonstrates strong generalization in open-domain text and long-context settings. This study is the first to empirically uncover and formalize a hybrid multi-mechanism retrieval paradigm, offering a novel foundation for interpretable modeling and robust reasoning in language understanding.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

A key component of in-context reasoning is the ability of language models (LMs) to bind entities for later retrieval. For example, an LM might represent"Ann loves pie"by binding"Ann"to"pie", allowing it to later retrieve"Ann"when asked"Who loves pie?"Prior research on short lists of bound entities found strong evidence that LMs implement such retrieval via a positional mechanism, where"Ann"is retrieved based on its position in context. In this work, we find that this mechanism generalizes poorly to more complex settings; as the number of bound entities in context increases, the positional mechanism becomes noisy and unreliable in middle positions. To compensate for this, we find that LMs supplement the positional mechanism with a lexical mechanism (retrieving"Ann"using its bound counterpart"pie") and a reflexive mechanism (retrieving"Ann"through a direct pointer). Through extensive experiments on nine models and ten binding tasks, we uncover a consistent pattern in how LMs mix these mechanisms to drive model behavior. We leverage these insights to develop a causal model combining all three mechanisms that estimates next token distributions with 95% agreement. Finally, we show that our model generalizes to substantially longer inputs of open-ended text interleaved with entity groups, further demonstrating the robustness of our findings in more natural settings. Overall, our study establishes a more complete picture of how LMs bind and retrieve entities in-context.

Problem

• Investigating how language models retrieve bound entities in context
• Identifying limitations of positional mechanisms with increased entity complexity
• Discovering lexical and reflexive mechanisms supplementing positional retrieval

Innovation

• LMs use positional, lexical, reflexive retrieval mechanisms
• A causal model combines three mechanisms for token prediction
• Model generalizes to longer inputs and natural settings