Definitional Sensitivity in Media Bias Detection: A Multi-Definition Dataset and Benchmark

2026-08-24

AI Summary

研究通过实验和模型评估，探讨了不同定义对媒体偏见标注的影响，发现定义的概念框架是导致标注差异的关键因素。

Abstract

Media bias detection relies on definitions and examples that specify what counts as bias, yet these specifications often vary across datasets or remain implicit, even when given the same name. Such variation makes it unclear whether models trained for the same bias category learn the same construct or different phenomena, a problem largely overlooked in prior work. We examine how definition choice affects bias annotation in a between-subjects experiment with 354 participants and a parallel evaluation with four LLMs. Participants and models rate six news articles across four bias categories using definitions that vary in conceptual framing and elaboration. Across 8,496 human and 28,800 LLM ratings, we find that the conceptual target of a definition drives annotation divergence, while construct-preserving elaboration does not: conceptual framing significantly shifts annotations for humans and does so even more strongly for LLMs. We discuss implications for construct specification in annotation protocols and prompt-based measurement, and consider how definitional sensitivity may propagate to downstream classification beyond media bias. We also release MUDD, the Multi-Definition Bias Detection Dataset.

Problem

• media bias detection
• definition variation
• annotation divergence
• conceptual framing

Innovation

• Definition Sensitivity
• Media Bias Detection
• Conceptual Framing
• Large Language Models (LLMs)
• MUDD Dataset