Beyond a Bag of Features: Set-Level Instability in Sparse Autoencoders

2026-08-11

AI Summary

This study investigates whether the activation sets of sparse autoencoders (SAEs) effectively capture human-like conceptual boundaries and typicality structures, and examines their mechanisms of semantic composition. We present the first systematic evaluation of SAEs for modeling human concepts, proposing to replace conventional cosine similarity with the overlap of activated feature sets as a measure of semantic similarity. Through controlled semantic perturbation experiments and comparisons with human typicality judgments, we find that SAE activation sets do not more accurately reflect human category boundaries or typicality than dense representations. Moreover, their responses to semantic changes significantly diverge from human judgments, suggesting that SAEs align more closely with internal model structure than with human semantics, thereby challenging the assumption of a “bag-of-features” compositional mechanism.

Abstract

Shani et al. (2026) show that LLM representations broadly recover human category boundaries, while failing to reflect fine-grained typicality structure. Their analysis uses cosine similarity over dense model representations. We revisit their approach using overlap over active sparse autoencoder (SAE) latent sets as a more interpretable similarity measure. We first verify that this set-level measure is meaningful: SAE latent sets can recover union-like compositional structure in controlled toy models and induce semantically coherent neighborhoods in natural text. Extending the human-concepts analysis to SAE set similarities, we find that SAE activation sets do not recover human category boundaries or within-category typicality more faithfully than dense embeddings or residual-stream states, but instead track model-internal similarity structure. To probe this gap further, we study active latent sets under well-controlled semantic modifications, revealing a substantial mismatch between human judgements of conceptual change and change in the SAE active set. We interpret this as evidence that, outside idealised settings, SAE features do not compose via simple bag-of-features semantics.

Problem

• sparse autoencoders
• concept representation
• human category boundaries
• typicality structure
• set-level similarity

Innovation

• sparse autoencoder
• set-level similarity
• compositional semantics
• interpretability
• concept representation