A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments

2025-09-29

AI Summary

While LLM-powered software agents are increasingly deployed in real-world decision-making domains (e.g., consumer choice, healthcare), existing evaluations predominantly assess task performance, neglecting whether agents exhibit human-like cognitive biases in their decisions. Method: We introduce ABxLab—the first open, behaviorally grounded evaluation framework for AI agent decision-making—employing controlled experiments in a simulated e-commerce environment to systematically manipulate variables such as price, user ratings, and psychological nudges, and quantitatively measure choice biases. Contribution/Results: Our experiments reveal that even without cognitive constraints, agents exhibit statistically significant, human-resembling preference biases—yet their underlying generative mechanisms differ fundamentally from human cognition. ABxLab establishes a novel paradigm for behavioral AI science and serves as a scalable, reproducible benchmark for evaluating agent decision quality beyond functional correctness.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Environments built for people are increasingly operated by a new class of economic actors: LLM-powered software agents making decisions on our behalf. These decisions range from our purchases to travel plans to medical treatment selection. Current evaluations of these agents largely focus on task competence, but we argue for a deeper assessment: how these agents choose when faced with realistic decisions. We introduce ABxLab, a framework for systematically probing agentic choice through controlled manipulations of option attributes and persuasive cues. We apply this to a realistic web-based shopping environment, where we vary prices, ratings, and psychological nudges, all of which are factors long known to shape human choice. We find that agent decisions shift predictably and substantially in response, revealing that agents are strongly biased choosers even without being subject to the cognitive constraints that shape human biases. This susceptibility reveals both risk and opportunity: risk, because agentic consumers may inherit and amplify human biases; opportunity, because consumer choice provides a powerful testbed for a behavioral science of AI agents, just as it has for the study of human behavior. We release our framework as an open benchmark for rigorous, scalable evaluation of agent decision-making.

Problem

• Studying AI agent decision-making in realistic consumer environments
• Assessing agent susceptibility to pricing and psychological influences
• Developing framework to evaluate behavioral biases in AI agents

Innovation

• Framework probes agent choice via attribute manipulation
• Web shopping environment tests price and nudge effects
• Open benchmark enables scalable agent decision evaluation