Through the LLM Looking Glass: A Socratic Self-Assessment of Donkeys, Elephants, and Markets

2025-03-20

AI Summary

This work addresses the challenge of detecting and quantifying implicit, subjective ideological biases—particularly in political and economic domains—within large language model (LLM) outputs. To overcome reliance on external annotations, we propose the novel “LLM Self-Socratic Reflective Evaluation” paradigm, enabling models to introspectively assess their own ideological leanings. We introduce two cross-cultural benchmark datasets: PoliGen (political bias) and EconoLex (economic bias), and design unsupervised, prompt-driven self-evaluation instructions. Extensive experiments across eight mainstream LLMs reveal consistent Democratic-leaning tendencies; Western models exhibit divergent economic stances, whereas Chinese models strongly favor socialist narratives. Our framework achieves the first stable, low-subjectivity quantification of latent systemic ideological bias. It establishes a new methodological foundation for LLM bias introspection and controllable, ideologically aware generation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

While detecting and avoiding bias in LLM-generated text is becoming increasingly important, media bias often remains subtle and subjective, making it particularly difficult to identify and mitigate. In this study, we assess media bias in LLM-generated content and LLMs' ability to detect subtle ideological bias. We conduct this evaluation using two datasets, PoliGen and EconoLex, covering political and economic discourse, respectively. We evaluate eight widely used LLMs by prompting them to generate articles and analyze their ideological preferences via self-assessment. By using self-assessment, the study aims to directly measure the models' biases rather than relying on external interpretations, thereby minimizing subjective judgments about media bias. Our results reveal a consistent preference of Democratic over Republican positions across all models. Conversely, in economic topics, biases vary among Western LLMs, while those developed in China lean more strongly toward socialism.

Problem

• Detecting subtle media bias in LLM-generated content
• Assessing LLMs' ability to identify ideological bias
• Measuring models' political and economic biases via self-assessment

Innovation

• Self-assessment to measure LLM biases directly
• Uses PoliGen and EconoLex datasets for evaluation
• Reveals Democratic and socialism preferences in LLMs