Super Weights in LLMs and the Failure of Selective Training

2026-07-09

AI Summary

This study investigates whether "super weights"—parameters deemed highly important in large language models—are genuinely suitable for selective training, challenging the assumption that parameter importance equates to trainability. Through a series of ablation experiments on OLMo-1B and OLMo-7B—including pruning analysis, local neighborhood training, random-position comparisons, and LoRA-based low-rank fine-tuning—the authors demonstrate for the first time that isolating and training only high-importance parameters leads to catastrophic performance degradation, collapsing to random-guess levels. In contrast, structured approaches such as LoRA or even training random subsets of parameters with equivalent capacity consistently outperform the baseline. These findings reveal that parameter importance does not imply trainability and underscore the critical role of structured parameter updates in efficient fine-tuning.

Abstract

Recent work identified Super Weights, individual parameters whose removal degrades model performance by orders of magnitude. We show that this degradation due to pruning Super Weights does not universally apply to all LLMs. Furthermore, if these parameters are so important, Super Weight-aware training should be effective. We show the opposite. Training Super Weights in isolation (100 to 8,192 parameters) drops accuracy to random-guessing levels on both OLMo-1B and OLMo-7B, and expanding to local neighborhoods of up to 36K parameters provides no improvement. The failure is specific to Super Weight coordinates: training an equal number of randomly chosen positions in the same down_proj layers instead improves over the baseline, so the collapse comes from targeting Super Weights, not from sparsity itself. Vanilla LoRA, updating every position in attention weight matrices through low-rank structure, succeeds with only 0.16% of parameters, and applying the same low-rank update to down_proj succeeds as well. A 10-seed ablation confirms that constraining LoRA updates at positions corresponding to Super Weight coordinates yields statistically indistinguishable results. These findings establish that parameter importance does not imply parameter trainability in isolation, and that effective fine-tuning relies on structured decompositions over entire layers rather than targeting individually important weights.

Problem

• Super Weights
• Large Language Models
• Parameter Trainability
• Selective Training
• Fine-tuning

Innovation

• Super Weights
• Selective Training
• LoRA
• Parameter Pruning
• Structured Fine-tuning