Cascaded Sparse Autoencoders Learn Multi-Level Visual Concepts in Multimodal LLMs

2026-06-15

AI Summary

This work addresses the lack of an interpretable, hierarchical conceptual structure in the visual representations of multimodal large language models (MLLMs), a limitation that existing sparse autoencoders (SAEs) struggle to resolve due to their inability to explicitly organize multi-level visual semantics. To overcome this, the paper proposes Cascaded Sparse Autoencoders (CSAE), which innovatively train a second-stage SAE directly on the decoder weights of a first-stage SAE, thereby learning higher-order “concepts of concepts” from lower-level features. This approach constructs a clear hierarchical visual concept system while avoiding prefix coupling in nested architectures and information bottlenecks in stacked designs. Experiments on mainstream MLLMs—including Qwen3-VL, Gemma-3, and LLaVA—demonstrate that CSAE significantly improves hierarchical concept consistency over existing SAE methods and yields concept groups amenable to effective population-level interventions, enhancing model controllability and interpretability.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated strong performance on vision-language tasks, yet their internal visual representations remain difficult to interpret. Sparse Autoencoders (SAEs) provide a scalable way to decompose dense model activations into sparse, interpretable features. However, existing SAE architectures primarily recover flat feature dictionaries and are less suited for explicit multi-level concept organization. In this paper, we introduce cascaded sparse autoencoders (CSAEs) for learning hierarchical visual concepts in MLLMs. Rather than nesting or stacking SAE sparse activation codes, CSAEs train a second-level SAE directly on the decoder weights of the first-level SAE, treating learned low-level feature directions as inputs for higher-level abstraction. This design enables CSAEs to learn "concepts of concepts" while avoiding drawbacks from the shared-prefix coupling of nesting, Matryoshka-style hierarchies and the bottlenecks of naively stacked SAEs. Experiments across Qwen3-VL, Gemma-3, and LLaVA on multiple visual datasets show that CSAEs improve interpretability in terms of hierarchical concept coherence over state-of-the-art SAE baselines. Results on concept steering further demonstrate that the learned concept groups support effective group-level interventions in MLLM outputs.

Problem

• Multimodal Large Language Models
• Interpretability
• Sparse Autoencoders
• Hierarchical Visual Concepts
• Visual Representation

Innovation

• Cascaded Sparse Autoencoders
• Hierarchical Visual Concepts
• Multimodal LLMs
• Interpretability
• Concept Steering