An Analysis of Active Learning Algorithms using Real-World Crowd-sourced Text Annotations

2026-04-25

AI Summary

This study addresses a critical limitation in conventional active learning, which assumes annotators are perfectly reliable—an assumption that rarely holds in real-world settings where labels are often noisy or missing. To bridge this gap, the authors present the first large-scale empirical evaluation of eight state-of-the-art deep active learning algorithms under realistic annotation conditions, using genuine crowdsourced text labels rather than synthetically injected noise. The assessment spans three standard datasets and systematically examines algorithmic robustness and practicality in the presence of both label noise and abstention (i.e., annotators declining to label). The findings reveal substantial performance disparities among existing methods when deployed in authentic environments, offering crucial guidance for real-world applications. To support further research, the authors publicly release their collected dataset of real human annotations.

Technology Category

• Machine Learning: Active Learning
• Humans and AI: Crowd Sourcing and Human Computation
• Natural Language Processing: Safety and Robustness

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Active learning algorithms automatically identify the most informative samples from large amounts of unlabeled data and tremendously reduce human annotation effort in inducing a machine learning model. In a conventional active learning setup, the labeling oracles are assumed to be infallible, that is, they always provide correct answers (in terms of class labels) to the queried unlabeled instances, which cannot be guaranteed in real-world applications. To this end, a body of research has focused on the development of active learning algorithms in the presence of imperfect / noisy oracles. Existing research on active learning with noisy oracles typically simulate the oracles using machine learning models; however, real-world situations are much more challenging, and using ML models to simulate the annotation patterns may not appropriately capture the nuances of real-world annotation challenges. In this research, we first collect annotations of text samples (from 3 benchmark text classification datasets) from crowd-sourced workers through a crowd-sourcing platform. We then conduct extensive empirical studies of 8 commonly used active learning techniques (in conjunction with deep neural networks) using the obtained annotations. Our analyses sheds light on the performance of these techniques under real-world challenges, where annotators can provide incorrect labels, and can also refuse to provide labels. We hope this research will provide valuable insights that will be useful for the deployment of deep active learning systems in real-world applications. The obtained annotations can be accessed at https://github.com/varuntotakura/al_rcta/.

Problem

• active learning
• noisy oracles
• crowd-sourced annotations
• text classification
• real-world annotation

Innovation

• active learning
• crowd-sourced annotations
• noisy oracles
• deep neural networks
• real-world evaluation