Slower Generalization, Faster Memorization: A Sweet Spot in Algorithmic Learning

2026-05-14

AI Summary

This work challenges the common intuition that more training data invariably accelerates generalization, investigating how dataset scale affects generalization and memorization dynamics in structured output tasks. Through controlled experiments on the Needleman–Wunsch matrix generation task using a small Transformer and a multiplication-based baseline, the study systematically analyzes validation accuracy and convergence behavior across varying data sizes. The findings reveal an optimal “sweet spot” in dataset scale that yields the fastest generalization; beyond this point, although the model still generalizes, it requires substantially more gradient updates, and the learning of underlying rules becomes decoupled from perfect training fit. This phenomenon is absent in the multiplication baseline, highlighting the distinct optimization dynamics inherent to Transformers.

Abstract

Critical-data-size accounts of grokking suggest a natural post-threshold intuition: once training data is sufficient to identify the underlying rule, additional data should accelerate validation convergence. We show that this intuition can fail in a controlled structured-output task. In Needleman--Wunsch (NW) matrix generation, small Transformers reach high validation exact-match accuracy fastest at an intermediate dataset size, not at the largest one. Past this dataset-size sweet spot, generalization remains achievable but requires more gradient updates. Conversely, in the regime where partial validation competence first appears, larger datasets can require fewer updates to reach high training accuracy, suggesting that emerging rule structure can accelerate fitting beyond example-wise memorization. A multiplication baseline does not show the same post-threshold slowdown. These results separate the critical data size for the onset of generalization from the dataset size that optimizes update-based convergence, and identify structured-output tasks where learning the rule and completing exact-fitting can diverge.

Problem

• grokking
• generalization
• structured-output
• dataset size
• memorization

Innovation

• grokking
• structured-output learning
• dataset size sweet spot
• generalization-memorization tradeoff
• Transformer convergence