A Detailed Factor Analysis for the Political Compass Test: Navigating Ideologies of Large Language Models

2025-06-24

AI Summary

Existing political alignment assessments—particularly the Political Compass Test (PCT)—are widely used to quantify the ideological positioning of large language models (LLMs), yet their validity and robustness remain unexamined. Method: We conduct controlled ablation experiments to systematically evaluate how prompt engineering, decoding parameters, and supervised fine-tuning affect PCT scores, while comparing the impact of politically skewed versus ideologically neutral training data. Contribution/Results: Prompt design and fine-tuning significantly alter PCT scores, whereas standard generation parameters exert negligible influence. Crucially, fine-tuning on highly politicized or deliberately neutral datasets yields no statistically significant change in PCT outcomes. These findings challenge the prevailing evaluation paradigm: PCT scores are highly sensitive to superficial interaction patterns rather than reflecting stable internal political representations, exposing fundamental structural limitations in using PCT as a valid ideological measurement instrument for LLMs.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Preferences

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Political Compass Test (PCT) or similar questionnaires have been used to quantify LLM's political leanings. Building on a recent line of work that examines the validity of PCT tests, we demonstrate that variation in standard generation parameters does not significantly impact the models' PCT scores. However, external factors such as prompt variations and fine-tuning individually and in combination affect the same. Finally, we demonstrate that when models are fine-tuned on text datasets with higher political content than others, the PCT scores are not differentially affected. This calls for a thorough investigation into the validity of PCT and similar tests, as well as the mechanism by which political leanings are encoded in LLMs.

Problem

• Analyze impact of generation parameters on LLM political scores
• Investigate external factors affecting model political leanings
• Assess validity of Political Compass Tests for LLMs

Innovation

• Analyze PCT scores with varied generation parameters
• Assess impact of prompt changes and fine-tuning
• Examine political content effect on fine-tuning