Cost-aware LLM-based Online Dataset Annotation

2025-05-21

AI Summary

To address the high computational overhead of majority voting and the difficulty of real-time accuracy–cost trade-offs in LLM-based online data annotation, this paper proposes CaMVo, a cost-aware online model selection framework. Methodologically, CaMVo introduces the first online adaptive LLM selection mechanism that requires neither pretraining nor ground-truth labels; it jointly leverages contextual bandits (LinUCB) and Bayesian confidence estimation to dynamically select an optimal subset of LLMs per input instance, with theoretically guaranteed lower bounds on accuracy. Furthermore, it enhances aggregation robustness via confidence-weighted voting. Experiments on MMLU and IMDB demonstrate that CaMVo reduces annotation cost substantially—achieving over 40% average inference cost savings—while matching or exceeding the accuracy of full-model voting. These results validate CaMVo’s efficiency and reliability in dynamic, resource-constrained annotation scenarios.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Recent advances in large language models (LLMs) have enabled automated dataset labeling with minimal human supervision. While majority voting across multiple LLMs can improve label reliability by mitigating individual model biases, it incurs high computational costs due to repeated querying. In this work, we propose a novel online framework, Cost-aware Majority Voting (CaMVo), for efficient and accurate LLM-based dataset annotation. CaMVo adaptively selects a subset of LLMs for each data instance based on contextual embeddings, balancing confidence and cost without requiring pre-training or ground-truth labels. Leveraging a LinUCB-based selection mechanism and a Bayesian estimator over confidence scores, CaMVo estimates a lower bound on labeling accuracy for each LLM and aggregates responses through weighted majority voting. Our empirical evaluation on the MMLU and IMDB Movie Review datasets demonstrates that CaMVo achieves comparable or superior accuracy to full majority voting while significantly reducing labeling costs. This establishes CaMVo as a practical and robust solution for cost-efficient annotation in dynamic labeling environments.

Problem

• Balancing cost and accuracy in LLM-based dataset annotation
• Reducing computational costs of majority voting in labeling
• Adaptive LLM selection for efficient online annotation

Innovation

• Adaptive LLM subset selection using contextual embeddings
• LinUCB-based mechanism for cost-confidence balance
• Weighted majority voting with Bayesian accuracy estimation