When Experimental Economics Meets Large Language Models: Tactics with Evidence

2025-05-27

AI Summary

Current LLM behavioral evaluation lacks standardized, reproducible experimental design guidelines. This paper systematically integrates principles from experimental economics into LLM research for the first time, proposing eight actionable experimental design strategies and establishing a methodological framework—Experimental Economics for LLMs (Eco-LLM). Two rigorously controlled, strategy-validated experiments demonstrate that Eco-LLM significantly enhances experimental rigor, result reproducibility, and cross-model comparability. The framework addresses a critical gap in LLM evaluation methodology and extends the theoretical scope and practical applicability of experimental economics in the AI era. By grounding LLM assessment in well-established behavioral and incentive-aligned experimental paradigms, Eco-LLM provides a generalizable methodological foundation for human-AI interaction studies, mechanism design in AI systems, and trustworthy AI evaluation in the digital age.

Technology Category

• Game Theory and Economic Paradigms: Mechanism Design
• Multiagent Systems: Mechanism Design
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Advancements in large language models (LLMs) have sparked a growing interest in measuring and understanding their behavior through experimental economics. However, there is still a lack of established guidelines for designing economic experiments for LLMs. By combining principles from experimental economics with insights from LLM research in artificial intelligence, we outline and discuss eight practical tactics for conducting experiments with LLMs. We further perform two sets of experiments to demonstrate the significance of these tactics. Our study enhances the design, replicability, and generalizability of LLM experiments, and broadens the scope of experimental economics in the digital age.

Problem

• Establish guidelines for economic experiments with LLMs
• Combine experimental economics and AI insights for LLM research
• Enhance design and replicability of LLM experiments

Innovation

• Combining experimental economics with LLM research
• Outlining eight practical LLM experiment tactics
• Enhancing design and replicability of LLM experiments