Enhancing Cost Efficiency in Active Learning with Candidate Set Query

2025-02-10

AI Summary

To address the high labeling cost and low efficiency in active learning (AL), this paper proposes a cost-effective classification AL framework. Methodologically, it introduces (1) a novel candidate-set querying paradigm that leverages dynamic conformal prediction to generate compact, reliable category candidate sets—significantly reducing the number of classes requiring human verification; and (2) a cost-sensitive acquisition function that jointly optimizes information gain and query cost. Evaluated on ImageNet64×64, the framework reduces labeling cost by 42% while maintaining competitive accuracy. Extensive experiments on CIFAR-10, CIFAR-100, and ImageNet64×64 demonstrate its effectiveness, robustness, and scalability. Notably, this work is the first to deeply integrate conformal prediction into the AL querying mechanism, establishing a new paradigm for low-cost, high-quality annotation.

Technology Category

• Machine Learning: Active Learning
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

This paper introduces a cost-efficient active learning (AL) framework for classification, featuring a novel query design called candidate set query. Unlike traditional AL queries requiring the oracle to examine all possible classes, our method narrows down the set of candidate classes likely to include the ground-truth class, significantly reducing the search space and labeling cost. Moreover, we leverage conformal prediction to dynamically generate small yet reliable candidate sets, adapting to model enhancement over successive AL rounds. To this end, we introduce an acquisition function designed to prioritize data points that offer high information gain at lower cost. Empirical evaluations on CIFAR-10, CIFAR-100, and ImageNet64x64 demonstrate the effectiveness and scalability of our framework. Notably, it reduces labeling cost by 42% on ImageNet64x64.

Problem

• Reduces labeling cost in active learning
• Narrows down candidate classes efficiently
• Uses conformal prediction for reliable candidate sets

Innovation

• Introduces candidate set query design
• Leverages conformal prediction for reliability
• Develops cost-effective acquisition function