Efficiently Representing Algorithms With Chain-of-Thought Transformers

2026-06-17

AI Summary

This work investigates whether Chain-of-Thought (CoT) Transformers can efficiently simulate algorithms formulated in the Word RAM model, circumventing the inefficiencies inherent in classical Turing machine simulations. Through rigorous theoretical analysis, the study establishes—for the first time—an efficient simulation relationship between CoT Transformers and the Word RAM model. It demonstrates that under three distinct settings—bounded precision, continuous inference, and a hybrid architecture combining linear RNNs with right-unique hard attention—any Word RAM algorithm can be simulated with only poly-logarithmic overhead, O(poly-log(n)), and even logarithmic overhead for specific instruction sets. This represents a substantial improvement over the quadratic overhead typical of Turing machine simulations and is successfully validated on canonical algorithms such as sorting and Dijkstra’s shortest-path algorithm.

Abstract

The increasing popularity of \emph{reasoning} models -- language models that output a series of reasoning or thought tokens before producing an answer -- is justified, in part, by theoretical results showing that chain-of-thought (CoT) transformers can simulate Turing machines, and thus perform arbitrary computation. However, the Turing machine, while suitable for complexity-theoretic analysis, is not convenient, intuitive, or efficient for discussing algorithms. Algorithms are typically designed and analyzed at a higher level of abstraction, captured by the \emph{Word RAM} model with random-access memory and unit-cost operations on $\bigO(\log n)$-bit words. As a result, Word RAM algorithms can be substantially more efficient than their Turing machine counterparts, raising the question: \emph{Can CoT transformers efficiently simulate Word RAM algorithms?} For instance, can they sort $n$ items in $\bigO(n \log n)$ steps or run Dijkstra's algorithm in $\bigO(E + V \log V)$ steps? We answer affirmatively, up to poly-logarithmic overhead. We first establish this for finite-precision transformers with poly-logarithmic width and rightmost unique hard attention, then strengthen the result to two more practical settings with finite width and log-precision: \emph{continuous} CoT, where reasoning takes the form of vectors rather than tokens, and a \emph{hybrid} architecture in which transformer layers sit atop a recurrent (linear RNN) layer. In all three cases, we find that CoT \emph{can} efficiently simulate any Word RAM algorithm with only a poly-logarithmic overhead in $n$. This overhead reduces to log-square when the Word RAM has a ``flat'' instruction set, and only logarithmic for multiplication-free flat instructions -- in stark contrast to known CoT simulations of Turing machines, which require quadratic overhead over Word RAM.

Problem

• Chain-of-Thought
• Word RAM
• Transformer
• algorithm simulation
• computational efficiency

Innovation

• Chain-of-Thought Transformers
• Word RAM
• efficient simulation
• poly-logarithmic overhead
• hybrid architecture