Active Learning with Task-Driven Representations for Messy Pools

2025-10-29

AI Summary

Active learning performance degrades significantly in uncurated data pools due to severe label noise and heterogeneous sample correlations. Method: This paper proposes a task-driven dynamic representation learning framework that, unlike existing approaches relying on fixed unsupervised or pretrained representations, periodically updates task-specific feature representations during active learning iterations—via semi-supervised learning or supervised fine-tuning—to better capture discriminative information. Contribution/Results: The core innovation lies in jointly optimizing representation learning and query selection, enabling a closed-loop evolution of representation–labeling–learning. Extensive experiments on multiple messy-pool benchmarks demonstrate that the proposed method consistently outperforms static-representation baselines, achieving simultaneous improvements in labeling efficiency and final model accuracy.

Technology Category

• Machine Learning: Active Learning
• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Active learning has the potential to be especially useful for messy, uncurated pools where datapoints vary in relevance to the target task. However, state-of-the-art approaches to this problem currently rely on using fixed, unsupervised representations of the pool, focusing on modifying the acquisition function instead. We show that this model setup can undermine their effectiveness at dealing with messy pools, as such representations can fail to capture important information relevant to the task. To address this, we propose using task-driven representations that are periodically updated during the active learning process using the previously collected labels. We introduce two specific strategies for learning these representations, one based on directly learning semi-supervised representations and the other based on supervised fine-tuning of an initial unsupervised representation. We find that both significantly improve empirical performance over using unsupervised or pretrained representations.

Problem

• Active learning struggles with messy uncurated data pools
• Fixed unsupervised representations fail to capture task-relevant information
• Task-driven representations need periodic updates during active learning

Innovation

• Active learning uses task-driven representations updated periodically
• Two strategies: semi-supervised learning and supervised fine-tuning
• Both methods outperform unsupervised or pretrained representations