On the Expressive Power of Transformers

2026-08-12

AI Summary

This work aims to delineate the expressive power of Transformers as language recognizers and theoretically align them with classical computational models. By leveraging circuit complexity theory, it establishes—for the first time—a systematic correspondence between architectural resources of Transformers (such as attention mechanisms and numerical precision) and parameters of Boolean circuits (including gate types, size, and depth), thereby formulating an analytical framework grounded in circuit complexity. The study precisely characterizes the classes of languages recognizable by Transformers under various configurations and identifies their respective positions within the circuit complexity hierarchy, offering a rigorous theoretical foundation for understanding the capabilities of large language models.

Abstract

Multi-layer transformers form the critical component of essentially all large language models (LLMs) in use today. Because of their ubiquity and computational capability, there is a rapidly growing body of work that aims to precisely calibrate the expressive power of transformers as language recognizers by comparing them against standard models of computation studied for decades by the theoretical computer science community. In this endeavor, circuit complexity has by and large emerged as the "correct" branch of computational complexity to analyze the expressive power of transformers; the reason is that parameterizing transformers by the various resources they use, such as attention and precision, leads to direct comparisons with different classes of circuits parameterized by resources such as type of gates, size, and depth. Here, we present an overview of selected results that delineate the expressive power of transformers using concepts and methods from circuit complexity.

Problem

• expressive power
• transformers
• language recognizers
• circuit complexity
• computational models

Innovation

• transformers
• expressive power
• circuit complexity
• computational complexity
• language recognizers