Feature Starvation as Geometric Instability in Sparse Autoencoders

2026-05-06

AI Summary

This work addresses the issue of feature starvation—manifested as dead neurons—and shrinkage bias in sparse autoencoders under overcomplete dictionaries, which stems from the ill-conditioned interplay between optimization dynamics and geometric structure. To overcome this, the authors propose the Adaptive Elastic Net Sparse Autoencoder (AEN-SAE), a fully differentiable architecture that integrates a strongly convex ℓ₂ regularization term with adaptive ℓ₁ reweighting. Theoretically grounded in Lipschitz stability and strong convexity guarantees, AEN-SAE enables end-to-end training without relying on heuristic resampling strategies. Empirical evaluations on both synthetic data and activations from large language models (Pythia 70M and Llama 3.1 8B) demonstrate that the method substantially alleviates feature starvation while preserving excellent reconstruction performance.

Abstract

Sparse autoencoders (SAEs) are used to disentangle the dense, polysemantic internal representations of large language models (LLMs) into interpretable, monosemantic concepts. However, standard $\ell_1$-regularized SAEs suffer from feature starvation (dead neurons) and shrinkage bias, often requiring computationally expensive heuristic resampling and nondifferentiable hard-masking methods to bypass these challenges. We argue that feature starvation is not merely an empirical artifact of poor data diversity, but a fundamental optimization-geometric pathology of overcomplete dictionaries: the $\ell_1$-induced sparse coding map is unstable and fundamentally misaligned with shallow, amortized encoders. To address this structural instability, we introduce adaptive elastic net SAEs (AEN-SAEs), a fully differentiable architecture grounded in classical sparse regression. AEN-SAEs combine an $\ell_2$ structural term that enforces strong convexity and Lipschitz stability with adaptive $\ell_1$ reweighting that eliminates shrinkage bias and suppresses spurious features, thereby jointly controlling the curvature and interaction structure of the induced polyhedral geometry. Theoretically, we show that AEN-SAEs yield a Lipschitz-continuous sparse coding map and recover the global feature support under mild assumptions. Empirically, across synthetic settings and LLMs (Pythia 70M, Llama 3.1 8B), AEN-SAEs mitigate feature starvation without auxiliary heuristics while maintaining competitive reconstruction abilities.

Problem

• feature starvation
• sparse autoencoders
• geometric instability
• shrinkage bias
• overcomplete dictionaries

Innovation

• feature starvation
• sparse autoencoders
• adaptive elastic net
• geometric instability
• Lipschitz stability