Polarization by Default: Auditing Recommendation Bias in LLM-Based Content Curation

2026-04-17

AI Summary

This study addresses the unclear nature of systemic biases in LLM-driven content recommendation and their impact on the information ecosystem, particularly the lack of cross-platform and prompt-strategy comparisons. Through a large-scale controlled simulation (540,000 top-10 recommendations) across OpenAI, Anthropic, and Google platforms—using real content from Twitter/X, Bluesky, and Reddit combined with six prompting strategies—the work provides the first systematic quantification of structural bias characteristics and prompt sensitivity regarding polarization, toxicity moderation, and sentiment orientation. Findings reveal that polarization is pervasive; toxicity moderation exhibits strategy-dependent reversals; sentiment bias is significantly negative; and left-leaning authors on Twitter/X are systematically over-recommended—a bias largely resistant to mitigation via prompt engineering.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Economics and fairness of platforms and recommendation systems
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

Large Language Models (LLMs) are increasingly deployed to curate and rank human-created content, yet the nature and structure of their biases in these tasks remains poorly understood: which biases are robust across providers and platforms, and which can be mitigated through prompt design. We present a controlled simulation study mapping content selection biases across three major LLM providers (OpenAI, Anthropic, Google) on real social media datasets from Twitter/X, Bluesky, and Reddit, using six prompting strategies (\textit{general}, \textit{popular}, \textit{engaging}, \textit{informative}, \textit{controversial}, \textit{neutral}). Through 540,000 simulated top-10 selections from pools of 100 posts across 54 experimental conditions, we find that biases differ substantially in how structural and how prompt-sensitive they are. Polarization is amplified across all configurations, toxicity handling shows a strong inversion between engagement- and information-focused prompts, and sentiment biases are predominantly negative. Provider comparisons reveal distinct trade-offs: GPT-4o Mini shows the most consistent behavior across prompts; Claude and Gemini exhibit high adaptivity in toxicity handling; Gemini shows the strongest negative sentiment preference. On Twitter/X, where author demographics can be inferred from profile bios, political leaning bias is the clearest demographic signal: left-leaning authors are systematically over-represented despite right-leaning authors forming the pool plurality in the dataset, and this pattern largely persists across prompts.

Problem

• recommendation bias
• polarization
• toxicity handling
• sentiment bias
• political leaning bias

Innovation

• recommendation bias
• large language models
• prompt sensitivity
• polarization
• content curation