CONSCIENTIA: Can LLM Agents Learn to Strategize? Emergent Deception and Trust in a Multi-Agent NYC Simulation

2026-04-10

AI Summary

This study investigates whether large language model (LLM) agents can spontaneously develop trust and deception strategies in multi-agent environments characterized by opposing incentives. To this end, we design a simplified New York City simulation where "blue" agents aim to reach their destinations efficiently, while hidden "red" agents profit by linguistically steering them toward routes dense with advertisements. Employing an iterative policy learning framework based on Kahneman-Tversky optimization (KTO), we present the first empirical evidence of strategic evolution among LLM agents in a controlled, large-scale multi-agent system. Our experiments demonstrate that blue agents improve their task success rate from 46.0% to 57.3%, indicating emergent selective cooperation; however, they remain susceptible to deceptive inducements 70.7% of the time, highlighting an inherent trade-off between safety and utility.

Technology Category

• Multiagent Systems: Mechanism Design
• Search and Optimization: Learning to Search
• Humans and AI: Learning Human Values and Preferences

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As large language models (LLMs) are increasingly deployed as autonomous agents, understanding how strategic behavior emerges in multi-agent environments has become an important alignment challenge. We take a neutral empirical stance and construct a controlled environment in which strategic behavior can be directly observed and measured. We introduce a large-scale multi-agent simulation in a simplified model of New York City, where LLM-driven agents interact under opposing incentives. Blue agents aim to reach their destinations efficiently, while Red agents attempt to divert them toward billboard-heavy routes using persuasive language to maximize advertising revenue. Hidden identities make navigation socially mediated, forcing agents to decide when to trust or deceive. We study policy learning through an iterative simulation pipeline that updates agent policies across repeated interaction rounds using Kahneman-Tversky Optimization (KTO). Blue agents are optimized to reduce billboard exposure while preserving navigation efficiency, whereas Red agents adapt to exploit remaining weaknesses. Across iterations, the best Blue policy improves task success from 46.0% to 57.3%, although susceptibility remains high at 70.7%. Later policies exhibit stronger selective cooperation while preserving trajectory efficiency. However, a persistent safety-helpfulness trade-off remains: policies that better resist adversarial steering do not simultaneously maximize task completion. Overall, our results show that LLM agents can exhibit limited strategic behavior, including selective trust and deception, while remaining highly vulnerable to adversarial persuasion.

Problem

• strategic behavior
• multi-agent simulation
• deception
• trust
• adversarial persuasion

Innovation

• multi-agent simulation
• strategic behavior
• Kahneman-Tversky Optimization
• emergent deception
• adversarial persuasion