Bridging Diversity and Uncertainty in Active learning with Self-Supervised Pre-Training

2024-03-06

arXiv.org

AI Summary

To address the challenge of harmonizing diversity and uncertainty sampling in active learning, this paper proposes TCM—a two-stage dynamic sampling framework tailored for self-supervised pretraining. In Stage I, TypiClust-based clustering ensures initial sample diversity to mitigate cold-start issues; in Stage II, margin-based uncertainty sampling is activated to enhance discriminative capability. Self-supervised pretraining strengthens representation robustness, while a learnable stage-switching mechanism enables data-volume-adaptive scheduling. Evaluated across multiple benchmark datasets, TCM consistently outperforms state-of-the-art methods under both low-resource (<10% labeled data) and high-resource (>50% labeled data) settings. It achieves superior sampling quality and generalization stability, offering a scalable, pretraining-aware paradigm for modern active learning.

Technology Category

• Machine Learning: Active Learning
• Search and Optimization: Sampling/Simulation-based Search
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

This study addresses the integration of diversity-based and uncertainty-based sampling strategies in active learning, particularly within the context of self-supervised pre-trained models. We introduce a straightforward heuristic called TCM that mitigates the cold start problem while maintaining strong performance across various data levels. By initially applying TypiClust for diversity sampling and subsequently transitioning to uncertainty sampling with Margin, our approach effectively combines the strengths of both strategies. Our experiments demonstrate that TCM consistently outperforms existing methods across various datasets in both low and high data regimes.

Problem

• Machine Learning
• Diversity in Selection
• Uncertainty Handling

Innovation

• TCM method
• Active Learning
• Pre-learning Framework