data labeling and annotation

Existing label-noise detection methods suffer from task specificity, limited learnability, and poor generalization across vision tasks. To address this, we propose the first unified, learnable framework for label-noise detection applicable to object detection, semantic segmentation, and instance segmentation. Our core innovation is an “error-correcting-via-error” paradigm: we reformulate label-noise detection as an instance segmentation problem by injecting controllable synthetic errors, and introduce a composite input representation alongside multi-task joint training. We establish a new authoritative benchmark on Cityscapes containing 459 real-world mislabeled instances. Extensive experiments demonstrate that our method significantly outperforms prior approaches, exhibiting strong generalization and robustness in both synthetic and real-world settings. This work provides a reproducible, scalable paradigm and empirical foundation for improving data quality in supervised learning.

High annotation costs and prolonged turnaround times plague NLP development, necessitating efficient and reliable data labeling paradigms. This paper proposes an LLM-powered Human-in-the-Loop (HITL) hybrid annotation framework that systematically integrates synthetic data generation, active learning, and human-AI collaboration, augmented with built-in mechanisms for annotation quality assessment, annotator management, and cost-benefit analysis. Unlike prior work—largely theoretical or narrowly scoped—this study introduces the first deployable, plug-and-play industrial-grade annotation methodology, bridging the critical gap between methodological research and real-world engineering practice. Empirical validation across multiple production NLP projects demonstrates that the framework consistently reduces annotation costs and cycle time by 30–50%, while maintaining label quality within required thresholds.

Widespread label noise (10–25%) in NLP benchmark datasets leads to systematic underestimation of model performance, with many purported “LLM failures” attributable to annotation errors rather than model limitations. Method: We propose LLM-as-a-judge—a framework leveraging ensemble judgments from GPT-4, Claude, and Llama, combined with consistency voting and error-sensitivity analysis to automatically detect mislabeled instances; we further apply label smoothing and confident learning for robust label recalibration. Contribution/Results: Comprehensive evaluation across the TRUE benchmark suite reveals substantial disparities in quality and efficiency among expert, crowdsourced, and LLM-generated annotations. After correction, state-of-the-art models achieve average accuracy gains of 3.2–7.8 percentage points. This work provides the first empirical evidence of systematic label-noise interference in LLM evaluation and introduces a scalable, collaborative adjudication paradigm that reframes data correction as model performance recalibration.

This work addresses the high cost of re-annotation in document layout analysis caused by evolving label categories by proposing a plug-and-play pseudo-labeling framework tailored for object detection. It introduces label propagation to document layout analysis for the first time, constructing multimodal object representations through the fusion of visual, textual, and positional embeddings. This enables efficient semi-supervised category propagation using only a small set of annotated samples. Experimental results on the D4LA dataset demonstrate that with merely 10% of the labeled data, the method achieves a mean average precision (mAP) of 54.0%, equivalent to 81.6% of the fully supervised performance, thereby substantially reducing manual annotation effort.

NLP data quality assessment has long relied on inter-annotator agreement, overlooking intra-annotator consistency—the temporal stability of individual annotators’ judgments. This neglect challenges the implicit “gold label as ground truth” assumption. Method: We conduct exploratory repeated annotation experiments across major NLP datasets and quantify intra-annotator agreement using Cohen’s and Fleiss’ Kappa, complemented by qualitative perceptual analysis. Contribution/Results: We demonstrate that mainstream NLP datasets routinely omit intra-annotator consistency reporting; moreover, individual annotators exhibit significant temporal variability in labeling identical texts. We identify and disentangle the dual influence of textual ambiguity and subjectivity on annotation stability. Our work establishes intra-annotator agreement as a foundational data quality metric, providing both a methodological framework and concrete guidelines for constructing more robust, reproducible NLP datasets.

Traditional machine learning treats systematic variations in human annotations—such as ambiguity, interpretive disagreement, and errors—as mere noise, thereby obscuring the true sources of error. This work reframes annotation as a measurement process and introduces the first statistical framework that simultaneously accommodates both shared and individualized notions of “ground truth.” The proposed model decomposes annotation variability into four interpretable components: instance difficulty, annotator bias, contextual noise, and relational consistency. Leveraging probabilistic modeling, we estimate and validate these components on multi-annotator natural language inference data. Empirical results confirm the presence of all four sources of variation and demonstrate that the framework effectively disentangles annotator behavior, offering actionable insights for constructing higher-quality datasets.

This study addresses the inconsistency in human annotation caused by ambiguous category definitions in traditional content moderation. To resolve this, the authors propose an AI-driven constitutional annotation framework: large language models first assist humans in formulating structured, interpretable category “constitutions,” which then guide automated dual-axis labeling of intent and content safety. This approach shifts human effort from case-by-case judgments to high-level semantic definition. Evaluated on harassment, hate speech, and non-violent criminal conduct tasks, the method reduces cross-model annotation inconsistency by up to 57-fold compared to conventional paragraph-based rules and effectively exposes latent gaps in existing policy formulations.

Manual annotation suffers from inconsistent quality and lacks systematic evaluation. Method: This paper proposes a consensus-based quality measurement framework grounded in multi-round annotation statistics, using dynamic decay of inter-annotator agreement variance as the core metric—established here as a “gold standard” for data quality. Recognizing annotators’ significant warm-up period but prohibitive cost of full-sample multiple annotation, we design a low-redundancy, high-efficiency progressive annotation protocol. The approach integrates statistical consistency analysis, variance convergence modeling, and label confidence estimation. Contribution/Results: Our paradigm substantially enhances data quality’s measurability and controllability: experiments show 3.2–7.8% accuracy gains across multiple NLP tasks and over 30% reduction in annotation redundancy.

This study addresses the prevailing tendency in machine learning to mischaracterize annotation disagreement as mere noise, thereby overlooking its value as a sociotechnical signal. Through a systematic literature review and reflexive thematic analysis of 346 papers from seven top-tier conferences (2020–2025), the work uncovers the mechanisms behind the “consensus trap” and its detrimental effects on algorithmic fairness. It critiques the “noise sensor” fallacy and advocates reinterpreting disagreement as a high-fidelity signal, proposing a new annotation paradigm centered on pluralistic experiential mappings rather than a singular “ground truth.” The analysis further reveals structural inequities—including the imposition of Western norms through geographic hegemony and annotators’ compliance driven by economic precarity—highlighting the erasure of positional visibility and the role of models as mediators of bias.

Label noise in real-world data severely degrades model performance, yet existing detection methods lack standardized, comparable evaluation protocols. To address this, we propose a three-component decomposition framework—comprising label consistency measurement, aggregation strategy, and information source selection—and establish the first unified cross-modal (image/tabular) benchmark for label noise detection. We introduce false negative rate at a fixed operating point as a fair, comparable metric. Extensive experiments systematically evaluate combinations of in-sample vs. out-of-sample information, average probability vs. majority voting aggregation, and logit margin vs. softmax confidence consistency measures, on both synthetic and real-world noisy datasets. Results demonstrate that in-sample information combined with average probability aggregation and logit margin-based consistency achieves superior performance across most settings. This work establishes the first interpretable, scalable, and empirically grounded evaluation framework for label noise detection, providing actionable insights for method selection and design.