Steering Vision-Language Models with Joint Sparse Autoencoders

2026-06-24

AI Summary

Existing sparse autoencoders struggle to produce interpretable representations in vision-language models that support controllable cross-modal interventions. This work proposes the Joint Sparse Autoencoder (JSAE), which explicitly enforces cross-modal alignment constraints to jointly decompose visual and linguistic activations into shared, interpretable image- and text-level features. By integrating sequence pooling with an additive intervention mechanism, JSAE enables, for the first time, bidirectional controllable interventions in vision-language models. Experiments demonstrate that JSAE successfully recovers semantically consistent cross-modal concepts in models such as LLaVA and uncovers asymmetric roles of different network layers in activation steering versus suppression, substantially enhancing the controllability and interpretability of internal representations at specific layers.

Abstract

Sparse Autoencoders (SAEs) have shown promise for analyzing language models, but applying them to vision-language models (VLMs) often yields representations that are difficult to use as controllable cross-modal steering directions. We introduce the Joint Sparse Autoencoder (JSAE), which uses an explicit alignment constraint to jointly factorize sequence-pooled vision and language activations into shared, interpretable image/caption-level features. Applied to LLaVA, JSAE recovers cross-modal features for recognizable concepts (e.g., food and animals). Through bidirectional interventions (additive steering and suppression), we observe a layer-dependent asymmetry under our protocol: additive steering peaks at mid-to-late (pre-output) layers and weakens at both ends, whereas suppression scores remain within a comparable range across all probed layers within statistical noise. Experiments on three VLMs, namely LLaVA-v1.6-Mistral-7B, Llama3-LLaVA-8B, and the MoE-based Qwen3-VL-30B, show related layer-localized effects across architectures. Together, these results suggest that explicitly aligned sparse representations support more controllable intervention-based analysis of multimodal features, within an identifiable layer range, than the unconstrained alternatives tested here.

Problem

• vision-language models
• sparse autoencoders
• cross-modal steering
• controllable representations
• multimodal features

Innovation

• Joint Sparse Autoencoder
• Vision-Language Models
• Cross-modal Steering
• Interpretable Representations
• Layer-dependent Intervention