Open Vocabulary Compositional Explanations for Neuron Alignment

2025-11-25

AI Summary

Existing neuron interpretation methods rely on human-annotated data and are constrained by predefined, domain-specific concepts. This paper addresses this limitation by proposing the first open-vocabulary compositional interpretation framework for vision, enabling semantic-spatial alignment between neurons and arbitrary visual concepts (e.g., “stripes”, “metallic texture”) or arbitrary image datasets—without requiring manual annotations. Our core method introduces open-vocabulary semantic segmentation into neuron interpretation: it leverages foundation models to generate concept-aligned masks and applies logical composition rules to infer spatial correspondences between neuron activation regions and open-ended concepts. Experiments demonstrate that our approach outperforms baselines both quantitatively and in human interpretability. It exhibits strong generalization across diverse tasks and multi-attribute scenarios, significantly enhancing the flexibility, scalability, and practical utility of neuron interpretation.

Technology Category

• Computer Vision: Interpretability, Explainability, and Transparency
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Neurons are the fundamental building blocks of deep neural networks, and their interconnections allow AI to achieve unprecedented results. Motivated by the goal of understanding how neurons encode information, compositional explanations leverage logical relationships between concepts to express the spatial alignment between neuron activations and human knowledge. However, these explanations rely on human-annotated datasets, restricting their applicability to specific domains and predefined concepts. This paper addresses this limitation by introducing a framework for the vision domain that allows users to probe neurons for arbitrary concepts and datasets. Specifically, the framework leverages masks generated by open vocabulary semantic segmentation to compute open vocabulary compositional explanations. The proposed framework consists of three steps: specifying arbitrary concepts, generating semantic segmentation masks using open vocabulary models, and deriving compositional explanations from these masks. The paper compares the proposed framework with previous methods for computing compositional explanations both in terms of quantitative metrics and human interpretability, analyzes the differences in explanations when shifting from human-annotated data to model-annotated data, and showcases the additional capabilities provided by the framework in terms of flexibility of the explanations with respect to the tasks and properties of interest.

Problem

• Overcoming reliance on human-annotated datasets for neuron explanations
• Enabling probing of arbitrary concepts in vision domain neurons
• Generating flexible explanations using open vocabulary segmentation

Innovation

• Open vocabulary semantic segmentation generates masks
• Framework probes neurons for arbitrary concepts
• Derives compositional explanations from model-annotated data