Analysis of Propaganda in Tweets From Politically Biased Sources

2025-07-10

AI Summary

This study investigates how mainstream journalists deploy propagandistic language on Twitter to disseminate political bias. We construct JMBX, the first annotated dataset comprising 1,874 tweets from 10 ideologically diverse media outlets spanning the far-left to far-right spectrum, enabling systematic analysis of individual-level propaganda usage in relation to outlet-level political orientation. We benchmark eight commercial large language models (LLMs) against a fine-tuned BERT baseline on propaganda detection, integrating human annotation with carbon footprint estimation tools to quantify both economic and environmental costs. Results show that LLMs achieve significantly higher accuracy than BERT but incur substantially greater per-instance inference cost—both financially and ecologically. Moreover, journalists from ideologically extreme outlets employ propaganda strategies more frequently. This work provides critical empirical evidence on the performance–sustainability trade-off in AI-powered content moderation, offering methodological innovation through integrated cost-aware evaluation and actionable insights for platform policy and sustainable AI deployment.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Large Multimodal Models (LMMs)
• Application Domains: Humanities & Computational Social Science

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

News outlets are well known to have political associations, and many national outlets cultivate political biases to cater to different audiences. Journalists working for these news outlets have a big impact on the stories they cover. In this work, we present a methodology to analyze the role of journalists, affiliated with popular news outlets, in propagating their bias using some form of propaganda-like language. We introduce JMBX(Journalist Media Bias on X), a systematically collected and annotated dataset of 1874 tweets from Twitter (now known as X). These tweets are authored by popular journalists from 10 news outlets whose political biases range from extreme left to extreme right. We extract several insights from the data and conclude that journalists who are affiliated with outlets with extreme biases are more likely to use propaganda-like language in their writings compared to those who are affiliated with outlets with mild political leans. We compare eight different Large Language Models (LLM) by OpenAI and Google. We find that LLMs generally performs better when detecting propaganda in social media and news article compared to BERT-based model which is fine-tuned for propaganda detection. While the performance improvements of using large language models (LLMs) are significant, they come at a notable monetary and environmental cost. This study provides an analysis of both the financial costs, based on token usage, and the environmental impact, utilizing tools that estimate carbon emissions associated with LLM operations.

Problem

• Analyzing journalist bias propagation via propaganda-like language in tweets
• Comparing LLM performance in detecting propaganda versus BERT-based models
• Assessing financial and environmental costs of LLM-based propaganda detection

Innovation

• Systematically collected annotated journalist tweet dataset
• Compared eight LLMs for propaganda detection
• Analyzed financial and environmental costs of LLMs