Weight-Adjusted Gradients Reveal Parameter Importance and Failure Modes in LLMs

2026-07-12

AI Summary

This work addresses the challenge that only a small subset of parameters in large language models disproportionately influences model performance, yet existing importance estimation methods struggle to identify them accurately. The authors propose Weight Adjustment Gradient (WAG), a novel approach that explicitly models the interaction between weights (zeroth-order) and gradients (first-order) to yield a more precise estimate of parameter importance. WAG demonstrates consistent effectiveness across diverse tasks—including expert allocation, parameter-level unlearning, mixed-precision quantization, and knowledge editing—by reliably pinpointing the minimal parameter subsets responsible for model collapse or dominant behaviors. Empirical results show that WAG significantly outperforms current importance metrics, offering a promising avenue to enhance model efficiency, reliability, and interpretability.

Abstract

Understanding which parameters are influential in Large Language Models (LLMs) is central to improving their efficiency, reliability, and interpretability. We introduce Weight-Adjusted Gradients (WAG), a simple yet effective approach for estimating parameter importance that explicitly captures the interaction between model weights and first-order gradient information and identifies parameters that disproportionately influence model behavior, such as those responsible for collapse phenomena in LLMs. Across a range of models and settings, we show that WAG surfaces a tiny but critical subset of parameters whose modification leads to dramatic degradation in performance, a failure mode that existing importance metrics overlook. These findings reveal a previously underexplored interplay between weights and gradients, suggesting that parameter importance cannot be fully understood through either signal alone. The surprising effectiveness of WAG points to fundamental structural properties of trained networks and motivates new open questions about the role of zeroth-order and first-order information in deep learning. We demonstrate the practical utility of WAG across multiple applications, including expert allocation in mixture-of-expert architectures, parameter-specific unlearning, mixed-precision quantization, and layer selection for knowledge editing. Our results position WAG as a unified approach for analyzing, debugging, and controlling LLMs, and opens new directions for principled model-level interpretation.

Problem

• parameter importance
• Large Language Models
• failure modes
• model interpretability
• weight-gradient interaction

Innovation

• Weight-Adjusted Gradients
• parameter importance
• large language models
• model interpretability
• failure modes