Universal computation is intrinsic to language model decoding

2026-01-12

AI Summary

This work investigates whether large language models possess universal computational capabilities and whether such capabilities arise from training or are inherent to their architecture. Through formal analysis of the autoregressive decoding mechanism, the study establishes—for the first time—that even randomly initialized, untrained language models are theoretically Turing-complete. This demonstrates that universal computation stems from intrinsic properties of the decoding process rather than being acquired through training. The research reframes the problem of computational controllability as one of prompt programmability, revealing that the primary role of training is to enhance the model’s ability to reliably and efficiently invoke its innate computational capacities via natural language prompts.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Language models now provide an interface to express and often solve general problems in natural language, yet their ultimate computational capabilities remain a major topic of scientific debate. Unlike a formal computer, a language model is trained to autoregressively predict successive elements in human-generated text. We prove that chaining a language model's autoregressive output is sufficient to perform universal computation. That is, a language model can simulate the execution of any algorithm on any input. The challenge of eliciting desired computational behaviour can thus be reframed in terms of programmability: the ease of finding a suitable prompt. Strikingly, we demonstrate that even randomly initialized language models are capable of universal computation before training. This implies that training does not give rise to computational expressiveness -- rather, it improves programmability, enabling a natural language interface for accessing these intrinsic capabilities.

Problem

• universal computation
• language models
• computational capabilities
• programmability
• autoregressive decoding

Innovation

• universal computation
• language models
• autoregressive decoding
• programmability
• random initialization