RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environments

2026-06-14

AI Summary

This study addresses the challenge that existing large language model (LLM) agents struggle to maintain coherent decision-making in dynamic, long-horizon retail environments. To this end, the authors construct a thousand-day simulation benchmark grounded in real-world supermarket operations, formalized as a partially observable Markov decision process (POMDP) that requires agents to jointly optimize pricing, replenishment, supplier selection, and shelf assortment. The work introduces the first economics-driven evaluation framework for long-horizon decision-making, integrating a data-driven simulation environment with a tool-augmented LLM architecture to systematically assess agents’ sustained reasoning and strategic consistency. Experimental results show that only a few models successfully complete the full 180-day evaluation, and even the best-performing LLM agent significantly underperforms an ideal policy in terms of net worth and sales, revealing critical limitations such as insufficient evidence gathering and short-sighted strategy formulation.

Abstract

Large language model (LLM) agents have made rapid progress on short-horizon, well-scoped tasks, yet their ability to sustain coherent decisions in dynamic long-horizon environments remains uncertain. We introduce RetailBench, a data-grounded simulation benchmark for evaluating tool-using LLM agents in single-store supermarket operation. RetailBench models retail management as a partially observable decision process and is designed to support thousand-day-scale simulations. In this environment, agents must manage pricing, replenishment, supplier selection, shelf assortment, inventory aging, customer feedback, external events, and cash-flow constraints. We evaluate seven contemporary LLMs under representative agent frameworks over a 180-day evaluation horizon and compare them with a privileged oracle policy. Results show substantial variation across models: only a small subset survives the full evaluation horizon, and even the strongest LLM runs remain substantially behind the oracle policy in final net worth and sales outcomes. Behavioral analysis attributes these gaps to incomplete evidence acquisition, surface-level decision making, and the lack of a consistent long-horizon policy. RetailBench provides a controlled testbed for studying reliable autonomy in economically grounded long-horizon decision-making.

Problem

• long-horizon reasoning
• coherent decision making
• LLM agents
• retail environment
• partially observable decision process

Innovation

• long-horizon reasoning
• LLM agents
• retail simulation
• partially observable decision process
• coherent decision making