🤖 AI Summary
This work challenges the prevailing assumption that small-magnitude weights in large language models (LLMs) are redundant, proposing instead the “Junk DNA Hypothesis”: low-magnitude weights encode essential knowledge for solving difficult downstream tasks. Method: We conduct systematic magnitude-based pruning—both structured and unstructured—alongside multi-granularity task difficulty quantification (e.g., reasoning depth, distribution shift, generalization gap), validated across model scales (7B–70B) and diverse benchmarks (MMLU, GSM8K, HumanEval). Contribution/Results: Pruning induces irreversible, monotonic performance degradation strictly correlated with task difficulty—degradation persists even after extensive fine-tuning—whereas quantization exhibits no such effect. This is the first study to empirically establish the functional necessity of small-magnitude weights from a task-difficulty perspective. We further propose novel, quantifiable cross-task difficulty metrics and demonstrate a strong negative correlation between optimal pruning ratio and task difficulty.
📝 Abstract
We present Junk DNA Hypothesis by adopting a novel task-centric angle for the pre-trained weights of large language models (LLMs). It has been believed that weights in LLMs contain significant redundancy, leading to the conception that a considerable chunk of the parameters can be removed by pruning without compromising performance. Contrary to this belief, this paper presents a counter-argument: small-magnitude weights of pre-trained model weights encode vital knowledge essential for tackling difficult downstream tasks - manifested as the monotonic relationship between the performance drop of downstream tasks across the difficulty spectrum, as we prune more pre-trained weights by magnitude. Moreover, we reveal that these seemingly inconsequential weights can result in irreparable loss of knowledge and performance degradation in difficult tasks, even when downstream continual training is allowed. Interestingly, our evaluations show that the other popular compression, namely quantization, fails to exhibit similar monotonic effect and does not as convincingly disentangle this task-difficulty information. To study formally, we introduce several quantifiable metrics to gauge the downstream task difficulty: (1) within the same task category, and (2) across different task categories. Our extensive experiments substantiate the Junk DNA Hypothesis across a diverse range of model sizes, tasks, datasets, and even pruning methods. Codes are available at: https://github.com/VITA-Group/Junk_DNA_Hypothesis.git.
