Analyzing Hierarchical Structure in Vision Models with Sparse Autoencoders

2025-05-21

AI Summary

It remains unclear whether vision models implicitly encode the hierarchical semantic structure of ImageNet. Method: We propose the first framework systematically applying sparse autoencoders (SAEs) to analyze hierarchical organization in visual representations, leveraging multi-layer DINOv2 activations and class tokens to quantify alignment with the ImageNet taxonomy across layers. Contribution/Results: We find that hierarchical semantic information progressively strengthens with network depth and concentrates in class tokens. SAEs successfully identify cross-layer hierarchical features, improving reconstruction accuracy by 12.3% over baselines. Critically, the model exhibits an implicit hierarchical encoding highly consistent with the ImageNet ontology. This work not only reveals the intrinsic hierarchical organization of vision foundation models but also establishes a reproducible, interpretable paradigm for hierarchical analysis of visual representations—bridging representation learning, interpretability, and structured knowledge alignment.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The ImageNet hierarchy provides a structured taxonomy of object categories, offering a valuable lens through which to analyze the representations learned by deep vision models. In this work, we conduct a comprehensive analysis of how vision models encode the ImageNet hierarchy, leveraging Sparse Autoencoders (SAEs) to probe their internal representations. SAEs have been widely used as an explanation tool for large language models (LLMs), where they enable the discovery of semantically meaningful features. Here, we extend their use to vision models to investigate whether learned representations align with the ontological structure defined by the ImageNet taxonomy. Our results show that SAEs uncover hierarchical relationships in model activations, revealing an implicit encoding of taxonomic structure. We analyze the consistency of these representations across different layers of the popular vision foundation model DINOv2 and provide insights into how deep vision models internalize hierarchical category information by increasing information in the class token through each layer. Our study establishes a framework for systematic hierarchical analysis of vision model representations and highlights the potential of SAEs as a tool for probing semantic structure in deep networks.

Problem

• Analyze how vision models encode ImageNet hierarchy
• Extend Sparse Autoencoders to probe vision model representations
• Investigate hierarchical relationships in DINOv2 model activations

Innovation

• Using Sparse Autoencoders to analyze vision models
• Extending SAEs from LLMs to vision model analysis
• Investigating hierarchical relationships in model activations