Enhancing LLM-Based Data Annotation with Error Decomposition

2026-01-17

AI Summary

This work addresses the instability of large language models (LLMs) in subjective annotation tasks—such as those involving psychological constructs—and the inadequacy of conventional alignment metrics in distinguishing error types and their downstream impacts. The authors propose a diagnostic evaluation framework that decouples LLM annotation errors along two dimensions: source (inherent task ambiguity versus model-induced error) and error type, introducing a novel error taxonomy tailored for ordinal subjective tasks. By integrating lightweight human audits, computational decomposition, and human–AI collaborative analysis, the approach is validated across four educational annotation tasks. It not only enables low-cost assessment of task suitability for LLM annotation but also offers actionable pathways for model refinement.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Large language models offer a scalable alternative to human coding for data annotation tasks, enabling the scale-up of research across data-intensive domains. While LLMs are already achieving near-human accuracy on objective annotation tasks, their performance on subjective annotation tasks, such as those involving psychological constructs, is less consistent and more prone to errors. Standard evaluation practices typically collapse all annotation errors into a single alignment metric, but this simplified approach may obscure different kinds of errors that affect final analytical conclusions in different ways. Here, we propose a diagnostic evaluation paradigm that incorporates a human-in-the-loop step to separate task-inherent ambiguity from model-driven inaccuracies and assess annotation quality in terms of their potential downstream impacts. We refine this paradigm on ordinal annotation tasks, which are common in subjective annotation. The refined paradigm includes: (1) a diagnostic taxonomy that categorizes LLM annotation errors along two dimensions: source (model-specific vs. task-inherent) and type (boundary ambiguity vs. conceptual misidentification); (2) a lightweight human annotation test to estimate task-inherent ambiguity from LLM annotations; and (3) a computational method to decompose observed LLM annotation errors following our taxonomy. We validate this paradigm on four educational annotation tasks, demonstrating both its conceptual validity and practical utility. Theoretically, our work provides empirical evidence for why excessively high alignment is unrealistic in specific annotation tasks and why single alignment metrics inadequately reflect the quality of LLM annotations. In practice, our paradigm can be a low-cost diagnostic tool that assesses the suitability of a given task for LLM annotation and provides actionable insights for further technical optimization.

Problem

• LLM-based data annotation
• subjective annotation
• error decomposition
• annotation ambiguity
• evaluation metrics

Innovation

• error decomposition
• subjective annotation
• diagnostic evaluation
• human-in-the-loop
• LLM alignment