Beyond Western Politics: Cross-Cultural Benchmarks for Evaluating Partisan Associations in LLMs

2025-09-23

AI Summary

This study systematically evaluates partisan association bias in large language models (LLMs) within non-Western sociopolitical contexts—particularly India—revealing risks of harmful, adversarial representations of political leaders and parties. Method: We propose a cross-cultural, fine-grained evaluation framework for partisan association that moves beyond Western-centric paradigms; construct two parallel, balanced question-answering datasets—NeutQA-440 (neutral) and AdverQA-440 (adversarial)—covering U.S. and Indian political entities; and quantify systemic bias across regimes via credibility judgment tasks and adversarial prompting. Contribution/Results: Experiments demonstrate significant pro-Democratic Party bias in U.S. contexts, while representations of India’s Bharatiya Janata Party exhibit mixed polarity but are overwhelmingly negative or neutral—confirming both asymmetry and structural nature of cross-cultural partisan bias in LLMs.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Large Multimodal Models (LMMs)
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Partisan bias in LLMs has been evaluated to assess political leanings, typically through a broad lens and largely in Western contexts. We move beyond identifying general leanings to examine harmful, adversarial representational associations around political leaders and parties. To do so, we create datasets extit{NeutQA-440} (non-adversarial prompts) and extit{AdverQA-440} (adversarial prompts), which probe models for comparative plausibility judgments across the USA and India. Results show high susceptibility to biased partisan associations and pronounced asymmetries (e.g., substantially more favorable associations for U.S. Democrats than Republicans) alongside mixed-polarity concentration around India's BJP, highlighting systemic risks and motivating standardized, cross-cultural evaluation.

Problem

• Evaluating partisan bias in LLMs beyond Western political contexts
• Examining harmful adversarial associations with political leaders and parties
• Creating cross-cultural benchmarks for comparative partisan bias assessment

Innovation

• Created adversarial and non-adversarial benchmark datasets
• Probed models for cross-cultural partisan associations
• Revealed systemic biases through comparative plausibility judgments