Problem
Research questions and friction points this paper is trying to address.
shopping agents
long-horizon tasks
hidden intent
benchmarking
requirement elicitation
EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent
🤖 AI Summary
This work addresses the limitations of existing shopping agent benchmarks, which fail to capture the realistic distribution of users’ hidden intentions across queries, profiles, and interactive clarifications, nor assess agents’ ability to uncover needs in long-horizon tasks. The authors introduce a new benchmark comprising 662 shopping tasks grounded in real Amazon products and reviews, requiring agents to infer and satisfy hidden user needs through up to 100 tool calls and ultimately recommend a single item. They explicitly decouple user intent into three components: observable queries, tool-gated profiles, and scripted clarifications, and propose a fine-grained scoring scheme with type- and source-aware labels to enable failure attribution. The dataset is generated via an automated pipeline with answers fixed prior to textual generation to ensure reliability. Evaluations on seven state-of-the-art models reveal that even the strongest achieves only 57.1% overall accuracy and performs significantly worse on hidden than explicit intents, exposing a critical bottleneck in current systems.
📝 Abstract
As LLM-based shopping agents enter production, existing benchmarks fail to capture how a shopper's requirements arrive: stated implicitly in the query, recorded in a profile, or revealed only when the right question is asked. Benchmarks that expose full intent upfront and grade only the final choice can neither pose this long-horizon challenge nor explain which requirement an agent missed. To address this gap, we introduce EComAgentBench, a benchmark of 662 tasks grounded in real Amazon products and reviews. Each task scatters these requirements across a visible query, a tool-gated profile, and scripted clarification; an agent must uncover hidden intent, verify candidates against attributes and review evidence, and commit to a single product within 100 tool calls. Moreover, typed, source-tagged rubrics grade every task, attributing each failure to a requirement and its source. Construction is automated yet reliable, with every answer fixed in code before any text is generated and every sample validated. Our evaluation of seven models reveals that even the strongest attains only 57.1% overall accuracy, and rubric satisfaction degrades from visible to hidden sources. Overall, we believe EComAgentBench will serve as a reproducible foundation for moving shopping agents from single-query search toward dependable assistance over long horizons.
Innovation
Methods, ideas, or system contributions that make the work stand out.
long-horizon tasks
hidden intent
shopping agents
rubric-based evaluation
tool-augmented reasoning
🔎 Similar Papers
No similar papers found.
