Task Decomposition for Efficient Annotation

2026-06-23

AI Summary

This work addresses the high cost and substantial cognitive load associated with structured annotation, which hinder efficient allocation of labeling efforts among heterogeneous annotators such as humans and models. The authors propose a center-theory-based task decomposition approach that identifies semantic centers to constrain the output space, formally models reasoning load, and introduces an algorithm for allocating heterogeneous annotation resources. Notably, this is the first method to integrate task decomposition with annotator capability matching. Experimental results demonstrate that the proposed approach significantly reduces cognitive burden while simultaneously improving both annotation quality and cost efficiency under a fixed budget.

Abstract

High-quality annotations of structured representations are expensive to collect over large corpora. Manual annotation of structure is laborious, and model-based annotation, although cheaper to generate, requires expensive validation and potentially significant supervision to ensure that the annotation quality is strong enough to be useful downstream. In traditional annotation workflows, annotation of each complete example is performed end-to-end by a single annotator. However, structured annotation is complex, and each aspect of the task represents a unique challenge with an associated inferential load for a given annotator. Modern annotation projects can incorporate heterogeneous groups of annotators, including both models and human annotators with varying domain and linguistic expertise. It remains unclear, however, how to redesign annotation tasks in this setting, where efforts are discriminately allocated across heterogeneous annotators with respect to distinct annotation challenges. We propose to decompose annotation tasks into sub-tasks in order to reduce the aggregate inferential load of annotation projects. Inspired by the notion of centers from centering theory, we introduce a formal model of inferential load based on the degrees of freedom in the space of valid annotations. Using this model, we show that identifying these centers (i.e. salient anchor entities realized by annotation sub-tasks) constrains the output space complexity, and decompositions which isolate and advance center identification reduce the aggregate inferential load. We provide guidelines for decomposing complex structured annotation tasks, supported by examples demonstrating improved cost-efficiency from our prior work. Finally, we present a procedure for allocating sub-tasks across annotators to maximize quality under a fixed budget.

Problem

• Task Decomposition
• Structured Annotation
• Inferential Load
• Annotation Efficiency
• Heterogeneous Annotators

Innovation

• task decomposition
• inferential load
• structured annotation
• centering theory
• annotation efficiency