A Mechanistic Understanding of Pronoun Fidelity in LLMs

2026-06-15

AI Summary

This work investigates the inconsistency in pronoun usage by large language models in multi-referent scenarios, which undermines the fairness and coherence of generated text. The study provides the first mechanistic account revealing that three distinct causal subspaces—group-entity binding, recency bias, and stereotype bias—coexist and compete within the model, collectively explaining 91–99.5% of pronoun generation behavior. Through unbounded distributed alignment search and attention head analysis, the authors identify the underlying routing mechanisms: group binding and stereotype bias share a concept-level localized routing pathway, whereas recency bias operates via a surface-form-dependent, distributed token-level routing strategy. This systematic dissection elucidates the internal mechanisms governing pronoun fidelity in large language models.

Abstract

Faithful and robust pronoun use is important for fair and coherent generations, yet large language models largely fail when multiple referents use different pronouns. To study the interplay of reasoning, repetition, and bias in this task, prior work relies exclusively on behavioural approaches, which may not reflect a model's internal workings. Therefore, we provide a mechanistic, model-internal perspective on pronoun fidelity, testing whether three mechanisms -- group entity binding (G), recency bias (R), and stereotypical bias (S) -- are causally implemented across several SOTA language models. Using Boundless Distributed Alignment Search, we find all three coexist as causal subspaces distributed across network depth. No single mechanism fully explains model behaviour, but a combination of the three consistently accounts for 91-99.5%. An attention head analysis further reveals two competing copying routes; group binding and stereotype share a localized concept-level route that retrieves a bound occupation-pronoun unit, while recency uses a distributed token-level route that repeats surface forms. In sum, pronoun fidelity arises from competition between simultaneously active causal subspaces.

Problem

• pronoun fidelity
• large language models
• referent resolution
• bias
• coreference

Innovation

• mechanistic interpretability
• pronoun fidelity
• causal subspaces
• distributed alignment search
• attention head analysis