Is More Data Worth the Cost? Dataset Scaling Laws in a Tiny Attention-Only Decoder

2026-04-10

AI Summary

This work addresses the critical challenge of efficiently balancing data scale against model performance under constrained computational and data resources. We propose an extremely minimalist attention-only decoder architecture and, under rigorously controlled experimental conditions, systematically investigate how training data volume influences model performance. For the first time, we empirically validate the existence of data scaling laws within a component-isolated, small-scale setting and quantify the diminishing marginal returns in performance gains as data size increases. Our experiments demonstrate that approximately 30% of the full training dataset suffices to achieve 90% of the validation accuracy obtained with the complete dataset, offering an effective data utilization strategy for resource-constrained environments.

Technology Category

• Machine Learning: Scalability of ML Systems
• Computer Vision: Large Vision Models
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Training Transformer language models is expensive, as performance typically improves with increasing dataset size and computational budget. Although scaling laws describe this trend at large scale, their implications in controlled, smaller-scale settings remain less explored. In this work, we isolate dataset-size effects using a strongly reduced attention-only decoder architecture. By training on progressively larger power-of-two subsets, we observe smooth performance improvements accompanied by clear diminishing returns, consistent with scaling-law behavior. Using only about 30% of the training data is sufficient to reach approximately 90% of the full-data validation token-level accuracy. These results provide actionable insights into dataset scaling in a controlled, component-isolated setting and offer practical guidance for balancing dataset size and computational cost in compute- and data-restricted environments, such as small research labs and exploratory model development.

Problem

• dataset scaling
• scaling laws
• data efficiency
• language models
• computational cost

Innovation

• dataset scaling laws
• attention-only decoder
• diminishing returns
• small-scale training
• computational efficiency