Do Language Models Track Entities Across State Changes?

2026-05-28

AI Summary

This study investigates how language models track entities in scenarios involving multiple state changes, with a focus on complex operations—such as PUT, REMOVE, and MOVE—that are poorly covered by existing evaluation benchmarks. Combining behavioral experiments, interpretability analyses, and intervention-based ablation studies, the work reveals that models employ a non-incremental strategy, aggregating information in parallel at the final token when the query is explicit. The research further uncovers that the REMOVE operation critically relies on fragile global inhibition signals, enabling the prediction and empirical validation of several failure modes, along with targeted mitigation strategies. These findings not only demonstrate the fundamentally non-sequential nature of how language models handle sequential tasks but also advance a synergistic paradigm integrating behavioral and mechanistic analysis.

Abstract

Entity tracking (ET), the ability to keep track of states, is a fundamental skill that underlies complex reasoning. An increasing amount of work investigates how transformer language models (LMs) solve entity binding $\textit{without}$ state changes. However, there is limited understanding of how non-toy LMs address ET problems of realistic difficulties expressed in natural language. To this end, we investigate the mechanisms underlying ET in more complex scenarios featuring multiple state-changing operations. We find that LMs do not incrementally track world states across tokens or query-relevant states across layers, but simply aggregate relevant information in parallel at the last token when the query becomes evident. We further investigate mechanisms of individual operations ($\texttt{PUT}$, $\texttt{REMOVE}$, $\texttt{MOVE}$) to characterize this non-incremental ET mechanism. Surprisingly, LMs implement the $\texttt{REMOVE}$ operation with a fragile global suppression tag; this global removal mechanism predicts various failure modes that we confirm behaviorally. We provide a mechanistic solution of nullifying this tag to partially address this issue. Overall, our findings reveal that LMs solve a fundamentally sequential task using a non-sequential strategy. More broadly, our work illustrates how behavioral and mechanistic analyses can fruitfully interact. Behavioral results inform mechanistic hypotheses, and insights from mechanistic analyses help build stronger behavioral evaluations by predicting failure modes missing from existing evaluations.

Problem

• Entity Tracking
• State Changes
• Language Models
• Natural Language Reasoning
• Transformer Models

Innovation

• Entity Tracking
• Language Models
• Mechanistic Interpretability
• State Changes
• Non-incremental Reasoning