Algebraic Decomposition Theory for Transformer Length Generalization

2026-08-13

AI Summary

This work addresses the lack of a precise characterization of Transformers’ length generalization capability on regular languages. By extending classical finite semigroup decomposition theory to the additive group of integers, the authors construct an algebraic framework based on the C-RASP formal system, thereby providing the first complete characterization of the class of regular languages amenable to length generalization by Transformers. This approach overcomes limitations of Krohn–Rhodes theory through the introduction of iterated cyclic decompositions and a polynomial-time decision algorithm. The resulting method not only efficiently determines whether any given regular language supports length generalization but also demonstrates significantly higher prediction accuracy than existing classification approaches in empirical evaluations.

Abstract

Transformer-based language models are known to sometimes generalize to sequences longer than seen during training, but we lack a precise characterization of which tasks admit length generalization. It is not even known which regular languages transformers length-generalize on -- and this is a foundational class of languages. Our contributions are to establish the first complete characterization of which regular languages transformers length-generalize on and provide a decision algorithm running in polynomial time in the size of the language's syntactic monoid. These results rely on an effective characterization of the regular languages in C-RASP, a recently-established formalism that expresses which languages transformers length-generalize on. This characterization is challenging because classical tools like Krohn-Rhodes decomposition theory for finite semigroups are insufficient for C-RASP. Firstly, the basic building blocks of Krohn-Rhodes theory -- flip-flop and simple groups -- are not expressible in C-RASP. Secondly, the basic building block of C-RASP (unbounded counting) is not expressible by the finite semigroups of Krohn-Rhodes theory. Thus, length generalization on regular languages is controlled by an algebraic property that is invisible to classical finite decomposition theory. We generalize classical decomposition theory from finite semigroups to the infinite additive group on the integers, allowing us to characterize C-RASP in terms of iterated wreath products of the integers and derive a provable polynomial-time decision algorithm for regular language membership. Experiments across a broad test suite of regular languages confirm that our theory captures transformers' length-generalization behavior more accurately than existing classifications.

Problem

• length generalization
• regular languages
• transformers
• algebraic decomposition
• C-RASP

Innovation

• length generalization
• algebraic decomposition
• C-RASP
• syntactic monoid
• wreath product