Small LLMs: Pruning vs. Training from Scratch

2026-06-12

AI Summary

This study systematically compares the performance of pruning large language models against training smaller models from scratch under an identical training token budget. Leveraging Llama-3.1-8B, the authors conduct controlled experiments using six structured and unstructured pruning strategies—spanning depth, width, and sparsity granularities—under strictly matched token conditions. The findings reveal that, with limited budgets, models initialized via pruning significantly outperform those with random initialization. As the training budget increases, the advantage of coarse-grained pruning diminishes and can be surpassed by training from scratch, whereas fine-grained pruning maintains a consistent performance edge. These results highlight the critical interplay between pruning granularity and training budget, offering new insights for efficient model compression.

Abstract

Pruning promises a shortcut to strong small language models. In this work, we examine this promise by pruning Llama-3.1-8B at pruning ratios of 0.5--0.8 with six methods spanning depth, width, and sparse granularities, under two controlled token-matched settings. (1) With the same training token budget, pruned initialization consistently outperforms random initialization. This shows that the parent model provides a strong starting point, although the advantage narrows as the training token budget grows and as the pruning ratio rises, nearly vanishing at the highest pruning ratio we study. (2) When training from scratch is instead given the full token budget consumed by the whole pipeline, pruning at finer granularities still retains an advantage, while coarser structured pruning can be matched or surpassed. This suggests that the parent model transfers knowledge that additional training tokens alone cannot fully recover, but only at fine granularity. Taken together, our results yield a clear recommendation: with a large pretrained model in hand and a limited training token budget, pruning is better than training from scratch; when the training budget is not limited, training from scratch can be competitive for coarser pruning, so a large pretrained parent is not always necessary.

Problem

• pruning
• training from scratch
• small language models
• token budget
• pretrained models

Innovation

• pruning
• training from scratch
• small language models
• token budget
• granularity