Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases

2026-05-19

AI Summary

This study investigates the “small-big data gap” phenomenon, wherein repeatedly training on a small dataset proves more computationally efficient than using a larger, non-repeated dataset. Through theoretical modeling and extensive empirical experiments across diverse tasks, architectures, and optimizers, the authors demonstrate that the sampling bias introduced by repeated data exposure acts as a beneficial inductive bias, promoting structured, layer-wise growth in neural networks and thereby accelerating convergence. This work is the first to attribute the computational efficiency of small-data repetition to this dynamic mechanism, challenging the conventional wisdom that “more data is always better.” It further establishes data repetition not merely as a constraint-driven compromise but as an active optimization strategy that consistently balances computational efficiency and training performance across multiple domains.

Abstract

This work investigates the ``small-vs-large gap'', where repeating on fewer samples can lead to compute saving during training compared to using a larger dataset. This is observed across algorithmic tasks, architectures and optimizers and cannot be explained using prior theory. We argue that the speedup comes from appropriate layer-wise growth enabled by sampling biases, which is more pronounced when the dataset size is smaller. We provide both theoretical analysis and empirical evidence from various interventions. Our results suggest that using a smaller dataset with more repetitions is not just a fallback strategy under data scarcity, but can be proactively leveraged as a favorable inductive biases for optimization, particularly in reasoning tasks.

Problem

• small-vs-large gap
• sampling biases
• dataset repetition
• training efficiency
• inductive bias

Innovation

• sampling bias
• layer-wise growth
• dataset repetition
• inductive bias
• training efficiency