A Circuit for Plural Reference: How LLMs Represent and Retrieve Singular and Plural Entities

2026-09-03

AI Summary

本文研究了大型语言模型如何表示和检索单复数实体以解决共指问题，通过机制解释性和注意力模式分析，发现了一组负责处理共指信息的注意力头。

Abstract

Coreference resolution is an important task in contextual reasoning. In this paper, we investigate the mechanism for representing and retrieving singular and plural entities for plural reference. We use a combination of mechanistic interpretability and attention pattern analysis to study the process in which LLMs predict a pronoun to refer back to previously mentioned entities. Using a range of causal intervention techniques, we find a set of attention heads that are responsible for (1) representing coreference information in the input, (2) identifying entities that form a plural reference, (3) transferring the information to the component that is responsible for selecting the antecedents and predicting the pronoun. We also find that LLMs align with humans in preference for plural pronoun. Specifically, entities in a plural construction are more likely to be referred to as a plural entity if they are ontologically similar and are linked by the conjunction "and".

Problem

• coreference resolution
• plural reference
• LLMs
• attention heads
• entity representation

Innovation

• mechanistic interpretability
• attention pattern analysis
• plural reference
• coreference resolution
• ontological similarity