Individual Parameters in Weight-Sparse Transformers Appear Interpretable

2026-07-03

AI Summary

This work proposes the first global interpretability framework based on weight-importance input sets, addressing the limitation of existing methods that often focus only on specific inputs or behaviors and fail to reveal the global functional roles of neural network weights. The framework identifies inputs critical to predictions and leverages automated large language models (LLMs) to generate concise, human-readable, and generalizable semantic descriptions of weight functions. In sparse Transformers, 12%–31% of weights admit such unambiguous semantic interpretations—significantly higher than in dense models. Ablation studies and cross-dataset validation further demonstrate that this advantage widens after filtering out unreliable descriptions, suggesting that sparse architectures inherently facilitate more structured and interpretable representations.

Abstract

A central goal of mechanistic interpretability is to understand how neural networks work and what each individual component does. Dominant circuit-finding approaches focus on a specific behavior and reverse-engineer the role of components on the associated sub-distribution. However, past work has shown that components can have different functions that are active on different subsets of the input distribution. In this work we ask whether a single weight can be understood globally across the full training distribution by characterizing when it matters (the inputs on which ablating it changes the model's predictions). We introduce an automated LLM pipeline that writes a short, human-readable description of when a weight matters and verifies it on held-out text, crediting a weight only if its description generalizes. Across two sparse and two dense transformers, the fraction of weights that are interpretable (in this sense) is higher in sparse transformers than in dense ones, a gap that widens once unreliable descriptions are discarded. Our results show that a meaningful fraction of a sparse transformer model's weights can be interpreted: 12 to 31% of weights have a single short description that identifies what the weight is used for.

Problem

• mechanistic interpretability
• weight interpretability
• sparse transformers
• model components
• input distribution

Innovation

• mechanistic interpretability
• weight sparsity
• automated interpretation
• transformer models
• global parameter understanding