Mechanistic Interpretability of Large-Scale Counting in LLMs through a System-2 Strategy

2026-01-06

arXiv.org

AI Summary

This work addresses the limitations of large language models (LLMs) in large-scale counting tasks, where performance degrades due to the depth constraints of the Transformer architecture. Inspired by System-2 cognition, the authors propose a test-time strategy that decomposes complex counting problems into independently solvable subproblems. Through observational and causal mediation analysis, attention head tracing, and representational probing, the study provides the first mechanistic account of System-2-like counting within LLMs, elucidating how counting information is stored, propagated, and aggregated across layers. This approach not only substantially improves counting accuracy on large-scale tasks but also overcomes inherent architectural limitations, offering both strong empirical performance and enhanced interpretability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Large language models (LLMs), despite strong performance on complex mathematical problems, exhibit systematic limitations in counting tasks. This issue arises from architectural limits of transformers, where counting is performed across layers, leading to degraded precision for larger counting problems due to depth constraints. To address this limitation, we propose a simple test-time strategy inspired by System-2 cognitive processes that decomposes large counting tasks into smaller, independent sub-problems that the model can reliably solve. We evaluate this approach using observational and causal mediation analyses to understand the underlying mechanism of this System-2-like strategy. Our mechanistic analysis identifies key components: latent counts are computed and stored in the final item representations of each part, transferred to intermediate steps via dedicated attention heads, and aggregated in the final stage to produce the total count. Experimental results demonstrate that this strategy enables LLMs to surpass architectural limitations and achieve high accuracy on large-scale counting tasks. This work provides mechanistic insight into System-2 counting in LLMs and presents a generalizable approach for improving and understanding their reasoning behavior.

Problem

• counting
• large language models
• architectural limitations
• transformer depth
• reasoning

Innovation

• mechanistic interpretability
• System-2 reasoning
• counting in LLMs
• attention heads
• test-time decomposition