Size Doesn't Matter: Cosine-Scored Sparse Autoencoders

2026-06-12

AI Summary

This work addresses a critical limitation in traditional sparse autoencoders, where inner-product-based scoring causes feature activations to be dominated by input norm rather than directional information—despite the fact that meaningful representations primarily rely on direction. Consequently, dictionary slots are often occupied by trivial norm detectors instead of semantically useful features. To resolve this, the paper introduces cosine similarity into sparse autoencoding for the first time and proposes a learnable scoring mechanism that adaptively weights cosine similarity and input norm per feature. This enables each feature to independently modulate its dependence on norm. Under identical reconstruction loss, the method substantially increases the number of human-interpretable semantic features while ensuring no feature becomes overly reliant on norm, thereby revealing the pivotal role of scoring geometry in achieving effective feature disentanglement.

Abstract

Sparse autoencoders (SAEs) detect features via inner product, so a feature's activation scales with both its directional alignment and the input's norm. Under BatchTopK, high-norm tokens inflate all pre-activations simultaneously, claiming dictionary slots regardless of content alignment. This matters because sublayer normalization has already discarded the magnitude the score measures, so the encoder detects a quantity the model does not read. We replace the score with a learned blend of cosine similarity and input magnitude, letting the optimizer choose how much norm to use; a per-feature extension lets each feature decide independently. In both regimes, training is free to recover inner product but never does, with no feature ever choosing more than half-magnitude dependence. At matched reconstruction, the cosine encoder learns features that align with human-recognizable concepts far more often than standard, filling dictionary slots that inner product wastes on norm detectors. Loss reweighting that equalizes gradients barely closes the gap, confirming forward-pass score geometry as the lever. The advantage is not universal across tasks or depths, but we believe cosine scoring should be the default for dictionary learning on normalized representations.

Problem

• sparse autoencoders
• feature activation
• cosine similarity
• input norm
• dictionary learning

Innovation

• cosine scoring
• sparse autoencoders
• feature alignment
• input norm
• dictionary learning