Revealing Political Bias in LLMs through Structured Multi-Agent Debate

2025-06-13

AI Summary

Large language models (LLMs) exhibit implicit political biases in socio-simulative contexts, yet their dynamic evolution mechanisms remain poorly understood. Method: We construct a structured multi-agent debate framework deploying LLM agents—parameterized with neutral, Republican, or Democratic identities and gender attributes—based on Llama, GPT, and Claude architectures. Agents engage in deliberations on sensitive political topics, with stance quantification performed via BERT-based stance detection. Contribution/Results: We first identify systematic Democratic-leaning bias in ostensibly neutral agents; observe Republican agents drifting toward neutrality through interaction; demonstrate gender-aware prompting triggers adaptive attitude shifts; and reveal that politically homogeneous agent groups rapidly form echo chambers reinforcing preexisting biases. This work establishes a reproducible behavioral benchmark for modeling political bias in LLMs, empirically characterizing directional stance drift, gender-mediated moderation effects, and structural conditions for echo chamber formation—thereby informing governance frameworks for trustworthy AI.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Humans and AI: Learning Human Values and Preferences

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions

Abstract

Large language models (LLMs) are increasingly used to simulate social behaviour, yet their political biases and interaction dynamics in debates remain underexplored. We investigate how LLM type and agent gender attributes influence political bias using a structured multi-agent debate framework, by engaging Neutral, Republican, and Democrat American LLM agents in debates on politically sensitive topics. We systematically vary the underlying LLMs, agent genders, and debate formats to examine how model provenance and agent personas influence political bias and attitudes throughout debates. We find that Neutral agents consistently align with Democrats, while Republicans shift closer to the Neutral; gender influences agent attitudes, with agents adapting their opinions when aware of other agents' genders; and contrary to prior research, agents with shared political affiliations can form echo chambers, exhibiting the expected intensification of attitudes as debates progress.

Problem

• Investigates political bias in LLMs using multi-agent debates
• Examines how LLM type and agent gender affect bias
• Analyzes echo chambers and attitude shifts in debates

Innovation

• Structured multi-agent debate framework
• Systematic variation of LLMs and genders
• Analysis of echo chambers in debates