Opponent Shaping in LLM Agents

2025-10-09

AI Summary

This work investigates whether large language model (LLM) agents possess opponent shaping (OS) capability—i.e., the ability to strategically influence other agents’ learning dynamics and decision preferences through iterative interactions—in multi-agent settings. To this end, we introduce OS to LLM agent research for the first time and propose ShapeLLM, a novel, model-free method specifically designed for Transformer architectures. ShapeLLM integrates game-theoretic environment modeling and enables gradient-free, multi-round strategic policy interaction. Experiments demonstrate that ShapeLLM successfully steers opponents toward exploitable equilibria in competitive scenarios and significantly improves collective payoff in cooperative tasks. These results empirically confirm that LLM agents exhibit bidirectional, dynamic behavioral shaping capacity. The study establishes a new paradigm for coordination and regulation in multi-agent LLM systems, advancing the understanding of strategic agency in foundation-model-based agents.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) are increasingly being deployed as autonomous agents in real-world environments. As these deployments scale, multi-agent interactions become inevitable, making it essential to understand strategic behavior in such systems. A central open question is whether LLM agents, like reinforcement learning agents, can shape the learning dynamics and influence the behavior of others through interaction alone. In this paper, we present the first investigation of opponent shaping (OS) with LLM-based agents. Existing OS algorithms cannot be directly applied to LLMs, as they require higher-order derivatives, face scalability constraints, or depend on architectural components that are absent in transformers. To address this gap, we introduce ShapeLLM, an adaptation of model-free OS methods tailored for transformer-based agents. Using ShapeLLM, we examine whether LLM agents can influence co-players' learning dynamics across diverse game-theoretic environments. We demonstrate that LLM agents can successfully guide opponents toward exploitable equilibria in competitive games (Iterated Prisoner's Dilemma, Matching Pennies, and Chicken) and promote coordination and improve collective welfare in cooperative games (Iterated Stag Hunt and a cooperative version of the Prisoner's Dilemma). Our findings show that LLM agents can both shape and be shaped through interaction, establishing opponent shaping as a key dimension of multi-agent LLM research.

Problem

• Investigating whether LLM agents can shape opponent learning dynamics
• Adapting opponent shaping methods for transformer-based language models
• Examining LLM influence on strategic behavior in game-theoretic environments

Innovation

• Adapted model-free opponent shaping for transformers
• Applied shaping across competitive and cooperative games
• Demonstrated LLM agents influence opponent learning dynamics