The Effect of State Representation on LLM Agent Behavior in Dynamic Routing Games

2025-06-18

AI Summary

This work addresses the problem of arbitrary and incomparable historical encoding in LLM-based agents operating in dynamic multi-agent games, stemming from their stateless nature. Methodologically, it introduces the first systematic natural-language “state representation” framework by decoupling and formalizing three key dimensions—action informativeness, reward informativeness, and prompt style—and proposes a prompt-engineering-based mechanism for history compression and reconstruction. Experiments are conducted in dynamic selfish routing games, involving multi-round interactions and quantitative evaluation of equilibrium deviation. Results demonstrate that summary-style history encoding, explicit regret signals, and constrained disclosure of others’ actions significantly improve alignment with and stability toward Nash equilibrium: behavioral variance decreases by 42%, and convergence stability is markedly enhanced. This work establishes a reproducible and comparable theoretical and practical foundation for state modeling in LLM agents.

Technology Category

• Multiagent Systems: Mechanism Design
• Search and Optimization: Learning to Search
• Game Theory and Economic Paradigms: Mechanism Design

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large Language Models (LLMs) have shown promise as decision-makers in dynamic settings, but their stateless nature necessitates creating a natural language representation of history. We present a unifying framework for systematically constructing natural language"state"representations for prompting LLM agents in repeated multi-agent games. Previous work on games with LLM agents has taken an ad hoc approach to encoding game history, which not only obscures the impact of state representation on agents' behavior, but also limits comparability between studies. Our framework addresses these gaps by characterizing methods of state representation along three axes: action informativeness (i.e., the extent to which the state representation captures actions played); reward informativeness (i.e., the extent to which the state representation describes rewards obtained); and prompting style (or natural language compression, i.e., the extent to which the full text history is summarized). We apply this framework to a dynamic selfish routing game, chosen because it admits a simple equilibrium both in theory and in human subject experiments cite{rapoport_choice_2009}. Despite the game's relative simplicity, we find that there are key dependencies of LLM agent behavior on the natural language state representation. In particular, we observe that representations which provide agents with (1) summarized, rather than complete, natural language representations of past history; (2) information about regrets, rather than raw payoffs; and (3) limited information about others' actions lead to behavior that more closely matches game theoretic equilibrium predictions, and with more stable game play by the agents. By contrast, other representations can exhibit either large deviations from equilibrium, higher variation in dynamic game play over time, or both.

Problem

• How state representation affects LLM agent behavior in dynamic routing games
• Systematic framework for constructing natural language state representations
• Impact of state representation on game theoretic equilibrium predictions

Innovation

• Unified framework for LLM state representation
• Summarized history improves equilibrium matching
• Regret info stabilizes agent behavior