ComboShoppingBench: Evaluating LLM Agents for Budget-Constrained Basket Shopping with Coupons

2026-08-10

AI Summary

Existing evaluation methods struggle to effectively assess large language models (LLMs) on multi-constraint combinatorial shopping tasks, which require simultaneous adherence to semantic plausibility and hard constraints—such as budget limits, coupon applicability, and item availability—and often admit multiple valid solutions. This work proposes the first evaluation framework that jointly accounts for semantic reasonableness and verifiable compliance with hard constraints. Leveraging a “witness basket” mechanism, the framework automatically generates user queries, constraint specifications, and evaluation criteria within simulated e-commerce and food-delivery environments. It combines semantic scoring with deterministic rule-based validation for comprehensive assessment. Experimental results reveal that state-of-the-art LLM agents perform poorly on this benchmark, highlighting significant deficiencies in their ability to perceive constraints and make reliable combinatorial decisions.

Abstract

Real-world shopping often requires constructing a basket of complementary items rather than retrieving a single product. Such combo-shopping tasks arise in device setup, meal preparation, event planning, and group takeout ordering, requiring joint reasoning about item compatibility, availability, store-level requirements, delivery fees, coupons, and budgets. Evaluation is challenging because multiple baskets may satisfy the same request, making exact-match metrics unsuitable, whereas semantic evaluation alone cannot detect infeasible orders, invalid coupon combinations, or incorrect payments. We introduce ComboShoppingBench, an agentic shopping benchmark for open-ended yet verifiable basket construction in a simulated commerce and takeout environment. During task synthesis, an exploration agent constructs a feasible and semantically coherent basket of purchasable products; this witness guides the generation of coupons, budget constraints, user queries, and aligned evaluation rubrics. During evaluation, LLM judges assess semantic satisfaction, response quality, and claim faithfulness, while deterministic validation checks product-ID validity, budget compliance, and coupon optimality. Experiments with diverse LLM agents demonstrate that even strong agents struggle on ComboShoppingBench, highlighting substantial room for improvement in reliable, constraint-aware combo shopping.

Problem

• combo shopping
• budget-constrained
• coupon optimization
• LLM evaluation
• shopping basket

Innovation

• combo shopping
• LLM agents
• budget-constrained optimization
• coupon reasoning
• verifiable evaluation benchmark