Decomposing The Dark Matter of Sparse Autoencoders

2024-10-18

arXiv.org

AI Summary

This work investigates the unreconstructed activation residuals—termed “dark matter”—in sparse autoencoders (SAEs), addressing their composition and interpretability bottlenecks. Methodologically, we integrate token-wise linear regression, cross-layer linear transformations, inference-time gradient tracing, and error-decomposition retraining. We formally introduce “introduced error” as a novel error category, systematically disentangling linear and nonlinear error components. Our key contributions are threefold: (1) dark matter is predominantly linearly predictable, yet residual nonlinear error constitutes the primary barrier to interpretability; (2) cross-layer linear transformations substantially reduce nonlinear error—outperforming gradient tracing—and this component contains fewer unlearned features; (3) nonlinear error exhibits stronger predictive power for SAE scaling behavior and contributes proportionally to downstream cross-entropy degradation, revealing fundamental differences in feature learning capacity, downstream impact, and scaling laws between linear and nonlinear error components.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Interpretability, Explainability, and Transparency
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Sparse autoencoders (SAEs) are a promising technique for decomposing language model activations into interpretable linear features. However, current SAEs fall short of completely explaining model performance, resulting in"dark matter": unexplained variance in activations. This work investigates dark matter as an object of study in its own right. Surprisingly, we find that much of SAE dark matter--about half of the error vector itself and>90% of its norm--can be linearly predicted from the initial activation vector. Additionally, we find that the scaling behavior of SAE error norms at a per token level is remarkably predictable: larger SAEs mostly struggle to reconstruct the same contexts as smaller SAEs. We build on the linear representation hypothesis to propose models of activations that might lead to these observations, including postulating a new type of"introduced error"; these insights imply that the part of the SAE error vector that cannot be linearly predicted ("nonlinear"error) might be fundamentally different from the linearly predictable component. To validate this hypothesis, we empirically analyze nonlinear SAE error and show that 1) it contains fewer not yet learned features, 2) SAEs trained on it are quantitatively worse, 3) it helps predict SAE per-token scaling behavior, and 4) it is responsible for a proportional amount of the downstream increase in cross entropy loss when SAE activations are inserted into the model. Finally, we examine two methods to reduce nonlinear SAE error at a fixed sparsity: inference time gradient pursuit, which leads to a very slight decrease in nonlinear error, and linear transformations from earlier layer SAE outputs, which leads to a larger reduction.

Problem

• Investigates unexplained variance in sparse autoencoder activations.
• Explores linear predictability of SAE error norms.
• Proposes methods to reduce nonlinear SAE error.

Innovation

• Linearly predict SAE dark matter from activations
• Analyze nonlinear SAE error characteristics
• Reduce nonlinear error via gradient pursuit