The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection

2024-11-17

arXiv.org

AI Summary

High-quality labeled data for media bias classification is costly and labor-intensive to obtain manually. Method: We propose the first LLM-driven synthetic data generation paradigm for this task, leveraging models such as GPT-4 and Claude to design zero-shot/few-shot annotation, instruction-tuning, and synthetic data cleaning pipelines, yielding Annolexical—a large-scale, high-quality synthetic dataset of 48K+ samples. Contribution/Results: We introduce the first framework for synthetic label quality validation and behavioral stress testing, systematically uncovering latent biases and trade-off boundaries in LLM-generated annotations. Experiments show that classifiers trained on LLM-annotated data outperform all annotating LLMs by 5–9 percentage points in Matthews Correlation Coefficient (MCC), match or exceed human-annotated baselines, and demonstrate strong cross-benchmark generalization on BABE and BASIL—while substantially reducing data curation costs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

High annotation costs from hiring or crowdsourcing complicate the creation of large, high-quality datasets needed for training reliable text classifiers. Recent research suggests using Large Language Models (LLMs) to automate the annotation process, reducing these costs while maintaining data quality. LLMs have shown promising results in annotating downstream tasks like hate speech detection and political framing. Building on the success in these areas, this study investigates whether LLMs are viable for annotating the complex task of media bias detection and whether a downstream media bias classifier can be trained on such data. We create annolexical, the first large-scale dataset for media bias classification with over 48000 synthetically annotated examples. Our classifier, fine-tuned on this dataset, surpasses all of the annotator LLMs by 5-9 percent in Matthews Correlation Coefficient (MCC) and performs close to or outperforms the model trained on human-labeled data when evaluated on two media bias benchmark datasets (BABE and BASIL). This study demonstrates how our approach significantly reduces the cost of dataset creation in the media bias domain and, by extension, the development of classifiers, while our subsequent behavioral stress-testing reveals some of its current limitations and trade-offs.

Problem

• Text Classification
• Data Annotation
• Media Bias

Innovation

• Large Language Models
• Media Bias Dataset
• Cost Reduction