Beyond the Hard Budget: Sparsity Regularizers for More Interpretable Top-k Sparse Autoencoders

2026-06-25

AI Summary

Top-k sparse autoencoders suffer from a fixed sparsity budget \(k\), which limits their adaptability to input complexity, often leading to overfitting and constraining feature interpretability and generalization. This work proposes introducing two soft sparsity regularizers prior to the Top-k selection: an \(\ell_1\) penalty applied to unselected units and a scale-invariant \(\ell_1/\ell_2\) ratio penalty restricted to units active within each batch. The approach demonstrates, for the first time, that hard Top-k sparsity and soft regularization can complementarily coexist—challenging conventional assumptions. Experiments across multiple datasets and vision foundation models show that the proposed method significantly enhances feature disentanglement without compromising reconstruction quality; moreover, the \(\ell_1/\ell_2\) regularizer concentrates activations into fewer latent units, improves robustness to variations in \(k\) during inference, and boosts linear probe performance under tight sparsity budgets.

Abstract

Sparse autoencoders (SAEs) have become a leading tool for interpreting the representations of vision foundation models, decomposing their polysemantic activations into a larger set of sparse, more monosemantic features. The Top-$k$ SAE, a now-standard variant, enforces sparsity architecturally through its activation function, retaining only the $k$ most active latents per input. Because it was designed precisely to avoid the $\ell_1$ penalty used by earlier SAEs and its known drawbacks, it has not been combined with an explicit sparsity regularizer, despite retaining limitations of its own, such as a budget $k$ that is fixed regardless of input complexity and a tendency to overfit to the training value of $k$. We introduce two sparsity regularizers compatible with the Top-$k$ architecture, both acting on the activations before the Top-$k$ selection: an $\ell_1$ penalty on the unselected (off-support) units, and a scale-invariant $\ell_1/\ell_2$-ratio penalty that concentrates the code onto fewer effective units. Both penalties are applied only to the batch-active units, those selected by the Top-$k$ operator at least once within the batch. Across two datasets, three vision foundation models, and a range of $k$, both regularizers consistently improve monosemanticity at no cost to reconstruction quality. The $\ell_1/\ell_2$ penalty further concentrates information into fewer latents, making reconstruction more robust to the inference-time choice of $k$ and improving small-budget linear probing. Our central finding is that hard architectural sparsity and soft sparsity regularization are complementary rather than mutually exclusive.

Problem

• Top-k sparse autoencoders
• sparsity regularization
• monosemanticity
• fixed sparsity budget
• overfitting to k

Innovation

• sparsity regularization
• Top-k sparse autoencoder
• monosemanticity
• L1/L2 ratio penalty
• interpretable representation