From Expressivity to Sample Complexity: Narrow Teachers for Transformers via C-RASP

2026-07-13

AI Summary

This study investigates the learnability and sample complexity of Transformer models under the assumption of sufficient expressive power. Focusing on narrow teacher models constructed from the C-RASP formal language, it extends existing expressivity analyses to the realm of learnability by integrating loss landscape analysis. The authors develop a theoretical framework tailored to C-RASP tasks and derive an upper bound on the sample complexity required to learn such structured functions. This work provides the first theoretical foundation for understanding the generalization capabilities of Transformers on structured tasks and establishes preliminary bounds on the associated sample complexity.

Abstract

A theoretical understanding of Transformers is crucial to better understand the capacities and limitations of large language models (LLMs). There is much work analyzing the expressivity of attention-based models. By proposing handcrafted weights or using computational complexity arguments, a large amount of past theoretical works have sought to characterize which tasks are and which are not in the hypothesis class of Transformer models. However, little work investigates the learnability of such solutions. In this work, we make progress towards this goal. Inspired by recent loss landscape analysis work, we propose preliminary sample complexity bounds for learning C-RASP constructions with Transformers.

Problem

• Transformers
• sample complexity
• learnability
• C-RASP
• theoretical understanding

Innovation

• sample complexity
• learnability
• Transformers
• C-RASP
• theoretical analysis