An evaluation of LLMs for political bias in Western media: Israel-Hamas and Ukraine-Russia wars

2026-01-05

arXiv.org

AI Summary

This study addresses the political bias prevalent in Western mainstream media coverage of international conflicts, which may distort public perception and undermine democratic processes. It presents the first systematic comparison of multiple large language models—including BERT, Gemini, and DeepSeek—in their ability to detect the editorial stance of The Guardian and BBC reports within the real-world contexts of the Russia–Ukraine war and the Israel–Hamas conflict, quantifying the distribution of left-leaning, right-leaning, and neutral tendencies. The findings reveal significantly greater volatility in media bias during the Israel–Hamas conflict compared to the Russia–Ukraine war. Moreover, systematic discrepancies emerge across models due to differences in architecture and training data: DeepSeek consistently exhibits a left-leaning orientation, whereas BERT and Gemini demonstrate greater neutrality. These results highlight how a model’s inherent “worldview” influences bias detection, offering both methodological foundations and empirical evidence for automated media bias assessment.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Bias, Fairness & Privacy

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Political bias in media plays a critical role in shaping public opinion, voter behaviour, and broader democratic discourse. Subjective opinions and political bias can be found in media sources, such as newspapers, depending on their funding mechanisms and alliances with political parties. Automating the detection of political biases in media content can limit biases in elections. The impact of large language models (LLMs) in politics and media studies is becoming prominent. In this study, we utilise LLMs to compare the left-wing, right-wing, and neutral political opinions expressed in the Guardian and BBC. We review newspaper reporting that includes significant events such as the Russia-Ukraine war and the Hamas-Israel conflict. We analyse the proportion for each opinion to find the bias under different LLMs, including BERT, Gemini, and DeepSeek. Our results show that after the outbreak of the wars, the political bias of Western media shifts towards the left-wing and each LLM gives a different result. DeepSeek consistently showed a stable Left-leaning tendency, while BERT and Gemini remained closer to the Centre. The BBC and The Guardian showed distinct reporting behaviours across the two conflicts. In the Russia-Ukraine war, both outlets maintained relatively stable positions; however, in the Israel-Hamas conflict, we identified larger political bias shifts, particularly in Guardian coverage, suggesting a more event-driven pattern of reporting bias. These variations suggest that LLMs are shaped not only by their training data and architecture, but also by underlying worldviews with associated political biases.

Problem

• political bias
• media analysis
• large language models
• Western media
• conflict reporting

Innovation

• political bias detection
• large language models
• media analysis
• event-driven reporting
• LLM comparison