Do Sparse Autoencoders Learn Meaningful Concept Hierarchies?

2026-06-22

AI Summary

Existing sparse autoencoders lack systematic evaluation of whether the features they learn form a meaningful hierarchy of generalization and specialization. This work formally defines, for the first time, the key semantic properties that an unsupervised concept hierarchy should satisfy and introduces a quantifiable evaluation framework grounded in semantic network and taxonomic theory. Through both quantitative and qualitative analyses, we demonstrate that while current sparse autoencoders can partially recover hierarchical structure, their performance is substantially limited by hard and soft feature absorption phenomena. Our study establishes new standards and provides empirical foundations for evaluating hierarchical organization in unsupervised concept discovery.

Abstract

Sparse autoencoders (SAEs) have become an important tool for unsupervised concept discovery in large models. To make the resulting feature spaces more interpretable and manageable, recent approaches have begun imposing hierarchical structure, either explicitly or as an implicit effect of training constraints, yet rigorous comparison remains difficult. There are no agreed-upon requirements for what a meaningful feature hierarchy should satisfy, and evaluation has largely relied on qualitative illustrations with fragmented quantitative protocols. To address this, we derive a set of key requirements for generalization/specialization hierarchies in unsupervised concept discovery, drawing on semantic net and taxonomy research alongside recent SAE work, and use them to derive a concrete evaluation protocol. Applying this protocol to current SAE approaches trained on visual data, we find that while feature spaces generally provide a basis for sensible hierarchies, establishing good hierarchical structure remains challenging. In particular, feature absorption, both in its well-known hard form and in a continuous, soft form, systematically compromises hierarchy quality, pointing to a fundamental tension that future approaches will need to navigate.

Problem

• Sparse Autoencoders
• Concept Hierarchies
• Unsupervised Concept Discovery
• Feature Hierarchy Evaluation
• Semantic Taxonomy

Innovation

• Sparse Autoencoders
• Concept Hierarchy
• Unsupervised Concept Discovery
• Feature Absorption
• Hierarchical Evaluation