Humans Hallucinate Too: Language Models Identify and Correct Subjective Annotation Errors With Label-in-a-Haystack Prompts

2025-05-22

AI Summary

Subjective NLP tasks—such as sentiment analysis and moral judgment—suffer from high inter-annotator disagreement, making it difficult to distinguish legitimate subjectivity from genuine annotation errors. Method: This paper introduces Label-in-a-Haystack, a novel paradigm, and the LiaHR framework, which treats discrepancies between large language model (LLM) outputs and original human labels as active correction signals—not merely noise. LiaHR integrates context-aware binary filtering, task-oriented instruction tuning, and structured prompting to jointly verify label credibility and perform automatic correction. Contribution/Results: Across multiple subjective benchmarks, LiaHR significantly improves accuracy and signal-to-noise ratio. Human evaluation and ecological validity analysis confirm that corrected labels preserve semantic plausibility and real-world applicability. The implementation is publicly available.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Active Learning
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Modeling complex subjective tasks in Natural Language Processing, such as recognizing emotion and morality, is considerably challenging due to significant variation in human annotations. This variation often reflects reasonable differences in semantic interpretations rather than mere noise, necessitating methods to distinguish between legitimate subjectivity and error. We address this challenge by exploring label verification in these contexts using Large Language Models (LLMs). First, we propose a simple In-Context Learning binary filtering baseline that estimates the reasonableness of a document-label pair. We then introduce the Label-in-a-Haystack setting: the query and its label(s) are included in the demonstrations shown to LLMs, which are prompted to predict the label(s) again, while receiving task-specific instructions (e.g., emotion recognition) rather than label copying. We show how the failure to copy the label(s) to the output of the LLM are task-relevant and informative. Building on this, we propose the Label-in-a-Haystack Rectification (LiaHR) framework for subjective label correction: when the model outputs diverge from the reference gold labels, we assign the generated labels to the example instead of discarding it. This approach can be integrated into annotation pipelines to enhance signal-to-noise ratios. Comprehensive analyses, human evaluations, and ecological validity studies verify the utility of LiaHR for label correction. Code is available at https://github.com/gchochla/LiaHR.

Problem

• Identifying subjective annotation errors in NLP tasks
• Correcting label errors using Large Language Models
• Improving signal-to-noise ratio in annotation pipelines

Innovation

• Uses LLMs for subjective label verification
• Introduces Label-in-a-Haystack prompting
• Proposes LiaHR framework for label correction