🤖 AI Summary
This work addresses the challenge of training highly efficient language models under stringent constraints—limited to a 16MB model size and 10 minutes of training time on 8×H100 GPUs—through a community-driven competition. The study proposes a comprehensive taxonomy encompassing 84 techniques across four categories: model architecture, training strategies, weight compression, and encoding optimization. Leveraging a large volume of participant submissions, it quantitatively evaluates the empirical contribution of each technique to bits-per-byte (BPB) performance and identifies methods with strong generalization capabilities. Over three competition phases, the best-reported BPB improved from 1.2244 to 1.058, reflecting a 13.6% gain and demonstrating that the cumulative effect of numerous small improvements can yield substantial overall performance enhancements.
📝 Abstract
How far can a language model improve under a strict artifact budget? Parameter Golf posed this question as an open community challenge in which participants trained the best language model, with the complete artifact (training code + compressed weights) required to fit within 16 MB and be trained in under ten minutes on 8xH100 SXM GPUs. Quality was measured in bits-per-byte (BPB), the average number of bits required to encode each byte of unseen text. We analyze 2,037 pull requests and 1,430 clean scored submissions from the contest, build a taxonomy of 84 optimization techniques, and measure each technique's contribution to BPB. The verified leaderboard score dropped from 1.2244 to 1.058 BPB across three phases -- a 13.6% reduction, despite individual techniques rarely improving BPB by more than 1%. We show that most gains in techniques shrink across competitive submissions, isolating the few methods that improve performance across stacks.