Efficient Human-in-the-Loop Active Learning: A Novel Framework for Data Labeling in AI Systems

2024-12-31

AI Summary

To address the low training efficiency of AI models caused by prohibitively high expert annotation costs (e.g., radiologist image interpretation), this paper proposes a human-in-the-loop active learning framework. Unlike conventional approaches that select only unlabeled samples, our framework innovatively integrates multiple query types—including sample selection, attribute queries, and explanatory feedback—within a unified, data-driven exploration-exploitation decision-making architecture. It combines Bayesian uncertainty estimation, multimodal query modeling, reinforcement learning–inspired policy optimization, and adaptive sampling to dynamically maximize information gain per expert interaction. Extensive experiments across five real-world datasets—including challenging medical imaging tasks—demonstrate that our method achieves an average 3.2% accuracy improvement over state-of-the-art baselines while reducing annotation cost by 27.5%.

Technology Category

• Machine Learning: Active Learning
• Humans and AI: Human-in-the-loop Machine Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Modern AI algorithms require labeled data. In real world, majority of data are unlabeled. Labeling the data are costly. this is particularly true for some areas requiring special skills, such as reading radiology images by physicians. To most efficiently use expert's time for the data labeling, one promising approach is human-in-the-loop active learning algorithm. In this work, we propose a novel active learning framework with significant potential for application in modern AI systems. Unlike the traditional active learning methods, which only focus on determining which data point should be labeled, our framework also introduces an innovative perspective on incorporating different query scheme. We propose a model to integrate the information from different types of queries. Based on this model, our active learning frame can automatically determine how the next question is queried. We further developed a data driven exploration and exploitation framework into our active learning method. This method can be embedded in numerous active learning algorithms. Through simulations on five real-world datasets, including a highly complex real image task, our proposed active learning framework exhibits higher accuracy and lower loss compared to other methods.

Problem

• Expert Time Efficiency
• AI Algorithm Performance
• Specialized Skill Domains

Innovation

• Active Learning Framework
• Dynamic Exploration-Exploitation
• Complex Image Tasks