🤖 AI Summary
The prevailing assumption that only Transformer architectures possess in-context learning (ICL) capability is challenged.
Method: We systematically evaluate standard multilayer perceptrons (MLPs) and MLP-Mixer variants on synthetic ICL benchmarks and classic psychological reasoning tasks—including Raven’s Progressive Matrices and analogical reasoning—under controlled computational budgets.
Contribution/Results: Contrary to conventional wisdom, both MLPs and MLP-Mixers achieve effective ICL, matching or exceeding Transformer performance on synthetic regression and classification tasks. Crucially, they significantly outperform Transformers on relational reasoning tasks, revealing previously underappreciated implicit relational modeling and structural generalization capabilities. This work provides the first empirical evidence that non-attention-based architectures can support robust ICL, thereby expanding the theoretical scope of ICL mechanisms and prompting a reevaluation of architectural priors in model selection.
📝 Abstract
In-context learning (ICL), the remarkable ability to solve a task from only input exemplars, is often assumed to be a unique hallmark of Transformer models. By examining commonly employed synthetic ICL tasks, we demonstrate that multi-layer perceptrons (MLPs) can also learn in-context. Moreover, MLPs, and the closely related MLP-Mixer models, learn in-context comparably with Transformers under the same compute budget in this setting. We further show that MLPs outperform Transformers on a series of classical tasks from psychology designed to test relational reasoning, which are closely related to in-context classification. These results underscore a need for studying in-context learning beyond attention-based architectures, while also challenging prior arguments against MLPs' ability to solve relational tasks. Altogether, our results highlight the unexpected competence of MLPs in a synthetic setting, and support the growing interest in all-MLP alternatives to Transformer architectures. It remains unclear how MLPs perform against Transformers at scale on real-world tasks, and where a performance gap may originate. We encourage further exploration of these architectures in more complex settings to better understand the potential comparative advantage of attention-based schemes.