PolBiX: Detecting LLMs' Political Bias in Fact-Checking through X-phemisms

2025-09-18

AI Summary

This study investigates political semantic bias-induced inconsistency in large language models’ (LLMs) fact-checking judgments. Addressing the gap in prior work—which predominantly examines macro-level ideological tendencies without fine-grained causal analysis—we propose an X-phemism substitution method based on German minimal opposing sentence pairs: systematically replacing euphemisms with dysphemisms (or vice versa) in claims while preserving factual content, thereby isolating and manipulating political affective load. Through controlled experiments and consistency evaluation across six state-of-the-art LLMs, we find that evaluative lexical choices significantly impair truth-value judgment accuracy—more strongly than latent political stance—and that prompt engineering emphasizing “objectivity” fails to mitigate this effect. To our knowledge, this is the first work to operationalize linguistic X-phemism for LLM bias detection, establishing a novel, interpretable, and controllable paradigm for assessing politically charged lexical bias.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Preferences

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

Large Language Models are increasingly used in applications requiring objective assessment, which could be compromised by political bias. Many studies found preferences for left-leaning positions in LLMs, but downstream effects on tasks like fact-checking remain underexplored. In this study, we systematically investigate political bias through exchanging words with euphemisms or dysphemisms in German claims. We construct minimal pairs of factually equivalent claims that differ in political connotation, to assess the consistency of LLMs in classifying them as true or false. We evaluate six LLMs and find that, more than political leaning, the presence of judgmental words significantly influences truthfulness assessment. While a few models show tendencies of political bias, this is not mitigated by explicitly calling for objectivism in prompts.

Problem

• Investigating political bias in LLMs' fact-checking consistency
• Assessing truthfulness variations through euphemism-dysphemism word exchanges
• Evaluating objectivism prompts' effectiveness in mitigating bias effects

Innovation

• Uses euphemism-dysphemism word exchange
• Constructs minimal claim pairs
• Evaluates six LLMs' classification consistency