Do Androids Dream of Unseen Puppeteers? Probing for a Conspiracy Mindset in Large Language Models

2025-11-05

AI Summary

This study investigates whether large language models (LLMs) exhibit conspiratorial thinking, harbor sociodemographic biases, and demonstrate susceptibility to prompt manipulation. Method: We introduce, for the first time in LLM evaluation, psychometrically validated scales—such as the Generic Conspiracist Beliefs Scale—into a rigorous assessment framework. Combining standardized questionnaires with multi-strategy prompt engineering (including role assignment, contextual priming, and sociodemographic attribute conditioning), we conduct cross-model, cross-prompt empirical probing across leading open- and closed-weight LLMs. Results: Unprompted, LLMs partially endorse conspiratorial statements; targeted prompts significantly amplify conspiratorial responses, with marked inter-model heterogeneity. Sociodemographic attribute conditioning induces systematic output skew, revealing latent biases and controllability risks. This work establishes a novel paradigm and empirical benchmark for cognitive modeling and safety evaluation of LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

In this paper, we investigate whether Large Language Models (LLMs) exhibit conspiratorial tendencies, whether they display sociodemographic biases in this domain, and how easily they can be conditioned into adopting conspiratorial perspectives. Conspiracy beliefs play a central role in the spread of misinformation and in shaping distrust toward institutions, making them a critical testbed for evaluating the social fidelity of LLMs. LLMs are increasingly used as proxies for studying human behavior, yet little is known about whether they reproduce higher-order psychological constructs such as a conspiratorial mindset. To bridge this research gap, we administer validated psychometric surveys measuring conspiracy mindset to multiple models under different prompting and conditioning strategies. Our findings reveal that LLMs show partial agreement with elements of conspiracy belief, and conditioning with socio-demographic attributes produces uneven effects, exposing latent demographic biases. Moreover, targeted prompts can easily shift model responses toward conspiratorial directions, underscoring both the susceptibility of LLMs to manipulation and the potential risks of their deployment in sensitive contexts. These results highlight the importance of critically evaluating the psychological dimensions embedded in LLMs, both to advance computational social science and to inform possible mitigation strategies against harmful uses.

Problem

• Investigating conspiratorial tendencies in Large Language Models
• Examining socio-demographic biases in conspiracy belief reproduction
• Assessing susceptibility to conditioning into conspiratorial perspectives

Innovation

• Administer psychometric surveys to LLMs
• Condition models with socio-demographic attributes
• Use targeted prompts to shift responses