Exemplars in Disguise: Pure Exemplar Models Mimic Abstraction-First Learning

2026-08-01

AI Summary

This study challenges the prevailing view that large language models acquire abstract knowledge before learning from individual instances. By constructing a purely memory-based generative model devoid of explicit abstract representations, and employing sensitivity analysis, input distribution modeling, and temporal evaluation, the authors demonstrate that current evaluation paradigms may produce misleading evidence of “abstraction-first” learning. They show that a model’s sensitivity to individual samples and the statistical properties of their input distribution can create the illusion of abstract reasoning—even when the model relies solely on memorized instances. The findings suggest that instance-specific and abstract knowledge may be inherently entangled within distributed representations, thereby undermining the empirical basis and theoretical assumptions underlying the abstraction-priority hypothesis.

Abstract

Whether idiosyncratic, item-specific knowledge is learned before abstract class-level generalizations, or vice versa, is a central question in language learning, with exemplar and abstraction-based theories making opposite predictions. Recent methods have claimed to show that, at least for large language models, abstract knowledge is learned first. We show that these methods fall short: pure memorizer models with no abstract representations can appear, by the same criteria, to learn either item-specific or class-level knowledge first, depending on their sensitivity to individual observations, with the transition point governed by the distributional properties of the input. We further argue that the distinction between item-specific and abstract knowledge may be ill-defined for distributed representations, as a word's class-level properties may not be separable from its item-specific properties.

Problem

• exemplar models
• abstraction
• language learning
• distributed representations
• item-specific knowledge

Innovation

• exemplar models
• abstraction-first learning
• distributed representations
• language learning
• memory vs generalization