Mix-Ecom: Towards Mixed-Type E-Commerce Dialogues with Complex Domain Rules

2025-09-28

AI Summary

Existing e-commerce agent benchmarks lack integrated evaluation capabilities for hybrid dialogue types and complex domain-specific rules. To address this, we propose Mix-Ecom—the first real-world customer service dialogue dataset incorporating four dialogue patterns, three core tasks, and 82 fine-grained e-commerce rules (4,799 samples). We further design a dynamic rule-aware reasoning framework, leveraging chain-of-thought annotation and post-hoc de-identification to ensure data quality and privacy compliance. Experiments reveal that mainstream large language model–based agents suffer from severe hallucination under complex rule constraints, and existing methods exhibit limited generalizability. Our framework significantly mitigates hallucination and improves task completion rates. The Mix-Ecom dataset will be publicly released to advance research on robustness and interpretability of e-commerce intelligent agents.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Data Mining & Knowledge Management: Rule Mining & Pattern Mining
• Multiagent Systems: Agent Communication

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

E-commerce agents contribute greatly to helping users complete their e-commerce needs. To promote further research and application of e-commerce agents, benchmarking frameworks are introduced for evaluating LLM agents in the e-commerce domain. Despite the progress, current benchmarks lack evaluating agents' capability to handle mixed-type e-commerce dialogue and complex domain rules. To address the issue, this work first introduces a novel corpus, termed Mix-ECom, which is constructed based on real-world customer-service dialogues with post-processing to remove user privacy and add CoT process. Specifically, Mix-ECom contains 4,799 samples with multiply dialogue types in each e-commerce dialogue, covering four dialogue types (QA, recommendation, task-oriented dialogue, and chit-chat), three e-commerce task types (pre-sales, logistics, after-sales), and 82 e-commerce rules. Furthermore, this work build baselines on Mix-Ecom and propose a dynamic framework to further improve the performance. Results show that current e-commerce agents lack sufficient capabilities to handle e-commerce dialogues, due to the hallucination cased by complex domain rules. The dataset will be publicly available.

Problem

• Evaluating agents' ability to handle mixed-type e-commerce dialogues
• Addressing limitations in handling complex e-commerce domain rules
• Benchmarking performance on multi-type dialogues across sales stages

Innovation

• Dynamic framework for mixed-type e-commerce dialogues
• Corpus with multiple dialogue types and domain rules
• Baselines addressing hallucination from complex rules