When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities

2026-07-09

AI Summary

This work addresses the challenge that sparse autoencoders in current vision-language models struggle to learn cross-modal consistent concepts, particularly suffering from fragmented visual representations. To overcome this, the authors propose the Structured Sparse Autoencoder (S²AE), which uniquely integrates semantic attention similarity with spatial proximity to group image patches. S²AE further introduces structured sparsity regularization—combining group sparsity and exclusive sparsity—to enforce intra-group conceptual consistency and inter-group disentanglement. Evaluated on Qwen2.5-VL-7B-Instruct, the method achieves a 6.06% improvement in mIoU, reduces the l₀ norm to 60.81, explains over 99% of variance, and enhances cross-modal semantic consistency and neuron monosemanticity by 3.08% and 2.37%, respectively.

Abstract

Sparse autoencoders (SAEs) have emerged as a promising technique for mechanistic interpretability by learning a set of sparse latent features in large models, each of which encodes a distinct concept. However, in vision-language models (VLMs), vanilla SAEs struggle to learn modality-consistent concepts, with concepts often exhibiting fragmented coverage (i.e., disjoint regions) in the visual modality. To address this challenge, we propose a Structured Sparse AutoEncoder ($S^2AE$) that enforces concept consistency from both semantic and spatial perspectives in the visual modality. Specifically, we group image patches based on Transformer attention similarity and spatial proximity, and introduce a structured sparsity regularization when training the vanilla SAE. The regularization consists of exclusive sparsity for inter-group concept disentanglement and group sparsity for intra-group concept consistency, which drives the latent neurons by SAEs to specialize in distinct, semantically grounded concepts. Evaluated on the \texttt{Qwen2.5-VL-7B-Instruct} model, the method achieves 6.06% average improvement in semantic alignment (mIoU) and 60.81 in representational efficiency (lower l0 norm) while maintaining near-perfect reconstruction fidelity with an Explained Variance above 99%. Cross-modal analysis further demonstrates that $S^2AE$ enhances neuronal monosemanticity by this visual structural prior, achieving a 3.08% average gain in semantic consistency and a 2.37% average gain in monosemanticity scores for both modalities of multimodal features, thereby fostering more coherent and disentangled representations.

Problem

• structured sparse autoencoders
• modality-consistent concepts
• vision-language models
• concept fragmentation
• cross-modal interpretability

Innovation

• Structured Sparse Autoencoder
• modality-consistent concepts
• structured sparsity regularization
• neuronal monosemanticity
• vision-language models