Contrastive Semantic Projection: Faithful Neuron Labeling with Contrastive Examples

2026-04-24

AI Summary

Existing neuron annotation methods rely on highly activating samples, often yielding overly broad or misleading labels that fail to accurately capture neuronal semantics. This work proposes Contrastive Semantic Projection (CSP), a novel approach that systematically incorporates contrastive samples—semantically similar yet low-activating—to enhance annotation quality through a two-stage pipeline. First, candidate labels are generated using a vision-language model; then, an improved CLIP-style encoder leverages contrastive sample pairs to refine and score these labels. Experiments demonstrate that CSP significantly outperforms existing baselines in both faithfulness and semantic granularity. Furthermore, case studies in tasks such as melanoma detection underscore the critical role of contrastive samples in elucidating neuron-level semantic representations.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Segmentation
• Machine Learning: Neuro-Symbolic Learning

Application Category

• Semantics and Knowledge: Representation, semantic annotation, enhancement, enrichments, access and/or integration of a variety of data on the Web
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Neuron labeling assigns textual descriptions to internal units of deep networks. Existing approaches typically rely on highly activating examples, often yielding broad or misleading labels by focusing on dominant but incidental visual factors. Prior work such as FALCON introduced contrastive examples -- inputs that are semantically similar to activating examples but elicit low activations -- to sharpen explanations, but it primarily addresses subspace-level interpretability rather than scalable neuron-level labeling. We revisit contrastive explanations for neuron-level labeling in two stages: (1) candidate label generation with vision language models (VLMs) and (2) label assignment with CLIP-like encoders. First, we show that providing contrastive image sets to VLMs yields candidate labels that are more specific and more faithful. Second, we introduce Contrastive Semantic Projection (CSP), an extension of SemanticLens that incorporates contrastive examples directly into its CLIP-based scoring and selection pipeline. Across extensive experiments and a case study on melanoma detection, contrastive labeling improves both faithfulness and semantic granularity over state-of-the-art baselines. Our results demonstrate that contrastive examples are a simple yet powerful and currently underutilized component of neuron labeling and analysis pipelines.

Problem

• neuron labeling
• contrastive examples
• faithfulness
• semantic granularity
• interpretability

Innovation

• contrastive examples
• neuron labeling
• vision-language models
• semantic projection
• faithful interpretation