Constant Bit-size Transformers Are Turing Complete

2025-05-22

arXiv.org

AI Summary

This work addresses the fundamental question of whether constant-bit-width Transformers can simulate arbitrary Turing machines and precisely characterizes their computational expressiveness. We introduce a novel simulation framework based on Post machines (queue automata), leveraging formal queue-behavior abstraction and rigorous computability-theoretic analysis. We establish, for the first time, that a constant-bit Transformer with context window length $ s(n) $ can simulate all computations in $ ext{SPACE}[s(n)] $, and achieves Turing completeness when $ s(n) = Omega(log n) $. Crucially, this result breaks the conventional assumption that expressive power necessitates scaling model parameters or numerical precision with input size. Our analysis yields the first exact equivalence between constant-bit Transformers and a classical space-complexity class, thereby providing a new theoretical foundation for understanding the intrinsic reasoning capabilities of Transformer architectures.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

We prove that any Turing machine running on inputs of arbitrary length can be simulated by a constant bit-size transformer, as long as the context window is sufficiently long. This improves previous works, which require scaling up either the model's precision or the number of parameters on longer inputs. Furthermore, we prove that the complexity class SPACE$[s(n)]$ exactly characterizes the expressive power of a constant bit-size transformer with a context window of length $s(n)$. Our approach relies on simulating Post machines, a Turing-complete computational model. Post machines can be modeled as automata equipped with a queue, exhibiting computational behaviors naturally aligned with those of transformers. The behavioral similarity between transformers and Post machines may offer new insights into the mechanisms underlying the reasoning abilities of transformers.

Problem

• Simulating Turing machines with constant bit-size transformers
• Characterizing transformer expressive power via SPACE complexity
• Modeling transformers through Post machines' computational behaviors

Innovation

• Constant bit-size transformers simulate Turing machines
• Uses Post machines with queue for computational alignment
• Exact characterization via SPACE complexity class