Sparse Autoencoders are Topic Models

2025-11-20

AI Summary

The interpretability of sparse autoencoders (SAEs) as topic models in embedding spaces remains contentious. This paper formally establishes SAEs as continuous-space topic models by interpreting their optimization objective as the maximum a posteriori (MAP) estimation of Latent Dirichlet Allocation (LDA) in the embedding space—thereby forging the first theoretical bridge between SAEs and classical probabilistic topic modeling. Building on this insight, we propose SAE-TM: a framework that explicitly interprets SAE latent features as semantically meaningful topics—rather than steering directions—for text and image modalities. SAE-TM enables zero-shot topic composition and cross-modal topic alignment without retraining. Empirically, on both textual and visual benchmarks, SAE-TM generates topics with superior coherence and diversity compared to strong baselines, all while requiring no additional training. Notably, it uncovers temporally evolving thematic patterns in a Japanese ukiyo-e image corpus, demonstrating its capacity for interpretable, modality-agnostic knowledge discovery.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Sparse autoencoders (SAEs) are used to analyze embeddings, but their role and practical value are debated. We propose a new perspective on SAEs by demonstrating that they can be naturally understood as topic models. We extend Latent Dirichlet Allocation to embedding spaces and derive the SAE objective as a maximum a posteriori estimator under this model. This view implies SAE features are thematic components rather than steerable directions. Based on this, we introduce SAE-TM, a topic modeling framework that: (1) trains an SAE to learn reusable topic atoms, (2) interprets them as word distributions on downstream data, and (3) merges them into any number of topics without retraining. SAE-TM yields more coherent topics than strong baselines on text and image datasets while maintaining diversity. Finally, we analyze thematic structure in image datasets and trace topic changes over time in Japanese woodblock prints. Our work positions SAEs as effective tools for large-scale thematic analysis across modalities. Code and data will be released upon publication.

Problem

• Sparse autoencoders are reinterpreted as topic models for embeddings
• SAE-TM framework learns reusable topics without retraining requirements
• Enables cross-modal thematic analysis in text and image datasets

Innovation

• Sparse autoencoders function as neural topic models
• SAE-TM framework learns reusable topic atoms
• Merges topic atoms without retraining for analysis