The Media Bias Detector: A Framework for Annotating and Analyzing the News at Scale

2025-09-29

AI Summary

This study systematically quantifies implicit bias in topic selection and framing across mainstream news media. We propose a scalable, multi-level bias analysis framework that integrates large language models with near-real-time news crawling to perform structured annotation—including political orientation, sentiment polarity, thematic classification, and event identification—on daily outputs from hundreds of outlets. Our methodology enables quantitative bias assessment at the sentence, article, and publisher levels, ensuring both precision and reproducibility. Since 2024, it has processed over 150,000 articles, uncovering cross-media systemic bias patterns; we concurrently release an open dataset and an interactive analytical platform. The core contribution is the first end-to-end, operational pipeline for measuring bias in large-scale, dynamic news streams—advancing media bias research from qualitative interpretation toward a verifiable, comparable, and accountable empirical paradigm.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Computer Vision: Bias, Fairness & Privacy
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Web Mining and Content Analysis: Sentiment analysis and opinion mining
• Security and Privacy: Large-scale security measurements

Abstract

Mainstream news organizations shape public perception not only directly through the articles they publish but also through the choices they make about which topics to cover (or ignore) and how to frame the issues they do decide to cover. However, measuring these subtle forms of media bias at scale remains a challenge. Here, we introduce a large, ongoing (from January 1, 2024 to present), near real-time dataset and computational framework developed to enable systematic study of selection and framing bias in news coverage. Our pipeline integrates large language models (LLMs) with scalable, near-real-time news scraping to extract structured annotations -- including political lean, tone, topics, article type, and major events -- across hundreds of articles per day. We quantify these dimensions of coverage at multiple levels -- the sentence level, the article level, and the publisher level -- expanding the ways in which researchers can analyze media bias in the modern news landscape. In addition to a curated dataset, we also release an interactive web platform for convenient exploration of these data. Together, these contributions establish a reusable methodology for studying media bias at scale, providing empirical resources for future research. Leveraging the breadth of the corpus over time and across publishers, we also present some examples (focused on the 150,000+ articles examined in 2024) that illustrate how this novel data set can reveal insightful patterns in news coverage and bias, supporting academic research and real-world efforts to improve media accountability.

Problem

• Measuring subtle forms of media bias at scale remains challenging
• Developing framework to study selection and framing bias systematically
• Providing empirical resources for analyzing media bias across publishers

Innovation

• Integrates large language models with real-time news scraping
• Quantifies media bias at sentence article and publisher levels
• Provides interactive web platform for data exploration