Length Generalization for Transformers via Compression

2026-09-08

AI Summary

该研究通过引入C-RASP+和C-RASP1片段，并采用压缩字符串方法，解决了变换器长度泛化的理论边界问题，提供了一个更紧致的样本大小界限。

Abstract

Recent advancements in transformer length generalization theory enable us to reliably predict when a transformer can learn to solve a task. In particular, the C-RASP hypothesis (a formalized version of the so-called RASP-l conjecture) posits that transformers length-generalize on a task if and only if a solution is expressible in the C-RASP language. While this hypothesis has strong empirical validation, theoretical problems arise from the fact that no computable length generalization bounds exist for C-RASP, alongside the discovery of seemingly contradictory experiments. To address these problems, we refine the C-RASP hypothesis utilizing the recently-proposed fragments C-RASP+ and C-RASP1. These fragments have computable length generalization bounds, though in the worst case requiring an extremely large (double exponential) sample size. It is an open question whether these sample size bounds are tight. In this paper, we resolve this open question by providing an exponentially tighter bound. In doing so, we show a polynomial length generalization bound for transformers if we adopt compressed strings, via a novel connection to power words. As an application, we show how this yields a fine-grained analysis of the C-RASP conjecture that resolves contradicting experimental evidence against it.

Problem

• length generalization
• transformers
• C-RASP hypothesis
• computable bounds
• contradictory experiments

Innovation

• C-RASP hypothesis
• length generalization bounds
• compressed strings
• power words