🤖 AI Summary
This work addresses the challenge of evaluating and selecting discriminative regions for fine-grained image classification in the absence of ground-truth labels. The authors propose a unified, CLIP-guided, unsupervised region scoring framework that systematically compares regions generated by SAM segmentation masks against those from random crops. They introduce two label-free pseudo-labeling variants based on global and local embeddings, respectively. By integrating multiple scoring strategies—including cosine similarity, margin-based boundary scores, entropy, and Soft Negative Margin—with a top-k region aggregation mechanism, they find that Soft Negative Margin yields the best performance, producing pseudo-labels nearly on par with true labels. Furthermore, random crops combined with a small top-k consistently outperform SAM-derived masks across all datasets, and the optimal aggregation strategy varies depending on the region generation method.
📝 Abstract
Recent vision models such as CLIP and SAM enable training-free segmentation and semantic encoding for fine-grained classification. A common approach is to compare the representations of segmented image regions with the text prompt embeddings of the corresponding labels. However, it remains unclear how different local regions and CLIP-based scoring strategies affect the selection of discriminative evidence, especially when ground-truth labels are unavailable. In this paper, we propose a unified CLIP-guided label-free region scoring framework for fine-grained classification. The framework evaluates cosine similarity-based, margin-based, and entropy-based scoring strategies using both SAM-generated masks and random crops, and introduces two label-free pseudo-label variants based on global image embeddings and local region embeddings. We conduct experiments on five fine-grained classification datasets to systematically compare different region generation methods and scoring strategies. The results show that Soft Negative Margin scoring achieves the strongest performance, and pseudo-label scoring closely approximates true-label performance. Although SAM produces semantically meaningful masks, random-crop-based pseudo-label scoring consistently outperforms SAM-based scoring across all datasets, suggesting that random crops preserve surrounding information and provide more stable semantic context when pseudo-labels are noisy. In addition, SAM masks benefit from aggregating embeddings from all regions, whereas random crops tend to perform better with a smaller top-k subset. These findings provide new insights for fine-grained classification.