QUORUM: QUality-Optimized Routing Using Multiple annotators

2026-08-28

AI Summary

本文提出QUORUM，通过动态分配实例给人类或大语言模型标注以优化质量并降低成本，利用特征信号估计实例难度，并通过基于一致性的奖励机制提高可靠性。

Abstract

Data annotation remains a central bottleneck in natural language processing, requiring human effort to obtain high-quality labels at scale. While Large Language Models (LLMs) offer a fast and cost-effective alternative, their reliability is highly instance-dependent: they perform well on simple inputs but often fail on examples requiring nuanced reasoning or contextual understanding. In this work, we address this challenge with QUORUM (QUality-Optimized Routing Using Multiple annotators), a budget-aware routing framework that dynamically assigns each instance to human or LLM annotators under a fixed annotation budget. Unlike prior approaches relying on model confidence or uncertainty estimates, QUORUM leverages feature-based signals to estimate instance difficulty and supports multiple annotations per instance, combining them through agreement-based rewards to improve reliability. We evaluate QUORUM across diverse closed- and open-ended annotation tasks in English and multilingual settings, and QUORUM improves annotation quality by up to 34.4% while reducing costs by 8.8% over competing methods. Code can be found at https://github.com/amazon-science/QUORUM.

Problem

• Data Annotation
• Large Language Models
• Instance-Dependent Reliability
• Natural Language Processing
• Annotation Quality

Innovation

• budget-aware routing
• feature-based signals
• multiple annotations
• agreement-based rewards