Testing the Assumptions of Active Learning for Translation Tasks with Few Samples

2026-04-10

AI Summary

This study investigates why active learning struggles to outperform random sampling in extremely low-resource neural machine translation settings with only 100–500 training samples. It systematically evaluates the core assumption underlying active learning—that informativeness and diversity effectively guide sample selection—under such data-scarce conditions. Through comprehensive experiments employing multiple active learning strategies, random baselines, and rigorously controlled variables, the work reveals for the first time that the optimization objectives of active learning exhibit no significant correlation with final test performance. Instead, the order of training samples and their interaction with pretraining data emerge as more critical factors influencing model effectiveness. These findings challenge prevailing active learning paradigms and suggest new directions for designing sampling strategies in few-shot machine translation scenarios.

Technology Category

• Machine Learning: Active Learning
• Search and Optimization: Sampling/Simulation-based Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Active learning (AL) is a training paradigm for selecting unlabeled samples for annotation to improve model performance on a test set, which is useful when only a limited number of samples can be annotated. These algorithms often work by optimizing for the informativeness and diversity of the training data to be annotated. Recent work found that AL strategies fail to outperform random sampling on various language generation tasks when using 100-500 samples. To understand AL's poor performance when only using few samples, we investigate whether the core assumptions underlying AL strategies hold. We find that neither the informativeness nor diversity of the training data, which AL strategies optimize for, are correlated with test set performance. Instead, factors like the ordering of the training samples and interactions with pre-training data have a larger impact on performance. This suggests that future AL methods must take these factors into account in order to work with very few samples.

Problem

• active learning
• few-shot learning
• machine translation
• sample selection
• pre-training

Innovation

• active learning
• low-resource translation
• sample informativeness
• training data diversity
• pre-training interaction