Counting as a minimal probe of language model reliability

2026-05-03

AI Summary

This study investigates whether the mathematical reasoning capabilities of large language models stem from genuine logical competence or merely from pattern matching. To isolate syntactic rule-following from semantic knowledge and ambiguity, the authors introduce a minimal probing task—stable counting—that requires models to accurately count repeated symbols until failure. This task eliminates confounding factors and directly assesses the reliability of rule execution. Through large-scale experiments on over 100 model variants, the work reveals for the first time that models implement counting mechanisms approximating finite-state machines, with stable counting capacities far below their nominal context lengths. Beyond this limited internal state capacity, rule adherence degrades to random guessing, demonstrating that current models lack robust and generalizable rule-following abilities.

Abstract

Large language models perform strongly on benchmarks in mathematical reasoning, coding and document analysis, suggesting a broad ability to follow instructions. However, it remains unclear whether such success reflects general logical competence, repeated application of learned procedures, or pattern matching that mimics rule execution. We investigate this question by introducing Stable Counting Capacity, an assay in which models count repeated symbols until failure. The assay removes knowledge dependencies, semantics and ambiguity from evaluation, avoids lexical and tokenization confounds, and provides a direct measure of procedural reliability beyond standard knowledge-based benchmarks. Here we show, across more than 100 model variants, that stable counting capacity remains far below advertised context limits. Model behavior is consistent neither with open-ended logic nor with stable application of a learned rule, but instead with use of a finite set of count-like internal states, analogous to counting on fingers. Once this resource is exhausted, the appearance of rule following disappears and exact execution collapses into guessing, even with additional test-time compute. These findings show that fluent performance in current language models does not guarantee general, reliable rule following.

Problem

• language model reliability
• rule following
• counting capacity
• procedural reliability
• logical competence

Innovation

• Stable Counting Capacity
• procedural reliability
• rule following
• language model evaluation
• internal states