Efficient Turing Machine Simulation with Transformers

2025-09-28

AI Summary

Existing constant-bit Transformer models simulating Turing machines (TMs) require Ω(s(n)) inference steps per TM step, resulting in suboptimal computational efficiency. Method: We introduce the multi-queue Turing machine (MQTM) as an intermediate model, prove its equivalence to multi-tape TMs, and design a synchronous bridging mechanism. Integrating sparse attention with fixed geometric-offset positional encoding, we construct the first constant-bit Transformer capable of efficiently simulating (t(n), s(n))-bounded TMs. Contribution/Results: Our model achieves O(s(n)^c) inference steps (for some c < 1) and O(s(n)) context length—nearly matching the theoretical lower bound. The core innovation lies in the synergistic co-design of MQTM-based modeling and lightweight attention, substantially narrowing the computational efficiency gap between neural architectures and Turing completeness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Constant bit-size Transformers are known to be Turing complete, but existing constructions require $Omega(s(n))$ chain-of-thought (CoT) steps per simulated Turing machine (TM) step, leading to impractical reasoning lengths. In this paper, we significantly reduce this efficiency gap by proving that any $(t(n),s(n))$-bounded multi-tape TM can be simulated by a constant bit-size Transformer with an optimal $O(s(n))$-long context window and only $O(s(n)^c)$ CoT steps per TM step, where $c>0$ can be made arbitrarily small by letting the Transformers'head-layer product sufficiently large. In addition, our construction shows that sparse attention with fixed geometric offsets suffices for efficient universal computation. Our proof leverages multi-queue TMs as a bridge. The main technical novelty is a more efficient simulation of multi-tape TMs by synchronous multi-queue TMs, improving both time and space complexity under stricter model assumptions.

Problem

• Reduces Transformer simulation steps for Turing machines efficiently
• Achieves optimal context window length for TM simulation
• Demonstrates sparse attention suffices for universal computation

Innovation

• Simulates Turing machines with optimal O(s(n)) context window
• Uses sparse attention with fixed geometric offsets
• Reduces CoT steps to O(s(n)^c) per TM step