S$^2$COPE: Self-Supervised Concept Discovery via Preference Learning

2026-06-12

AI Summary

This work addresses the tension between the uninterpretable nature of self-supervised learning features and the reliance of interpretable models on manual annotations by proposing the first fully unsupervised, end-to-end framework that leverages a vision-language foundation model as an active agent. Through a self-supervised preference optimization loop, the model autonomously hypothesizes, validates, and reinforces candidate visual attributes directly from raw images, enabling the emergence of structured conceptual representations without any human-provided labels. This approach achieves interpretable concept discovery solely through autonomous interaction between the model and data, internalizing these concepts into the model’s backbone. Evaluated on downstream classification tasks across natural, medical, and physical domains, the method improves Top-1 accuracy by up to 24 percentage points, substantially outperforming standard vision-language models.

Abstract

Current representation learning paradigms force a fundamental compromise: self-supervised methods scale to massive datasets but yield opaque features, whereas interpretable models remain bottlenecked by the need for dense human annotation. We introduce Self-Supervised Concept discOvery via Preference lEarning (\model), a label-free framework that resolves this dilemma. Instead of treating Vision-Large-Language Models (VLLMs) as static feature extractors, \model leverages them as active participants in a self-supervised preference optimization loop. By autonomously hypothesizing, validating, and reinforcing candidate visual attributes directly from raw imagery, our framework discovers novel, structured concepts without a single label. Extensive experiments across natural, medical, and physics domains demonstrate that \model successfully extracts domain-specific concepts where standard VLLMs often fail to generate. By amortizing concept discovery directly into the VLLM backbone through our self-supervised preference objective -- rather than relying on static generation and disjoint filtering -- we achieve up to a 24-point absolute improvement in downstream top-1 classification accuracy on unseen data. Our work suggest that interpretability can emerge through a model's autonomous interaction with incidental visual structures, without any human supervision.

Problem

• self-supervised learning
• concept discovery
• interpretability
• preference learning
• vision-language models

Innovation

• self-supervised learning
• concept discovery
• preference learning
• vision-language models
• interpretability