ChineseEcomQA: A Scalable E-commerce Concept Evaluation Benchmark for Large Language Models

2025-02-27

AI Summary

Existing LLM evaluation benchmarks for e-commerce suffer from insufficient coverage of task heterogeneity, ambiguity between general and domain-specific capabilities, and difficulties in verifying factual accuracy. This paper introduces ChineseEcomQA, a novel Chinese e-commerce conceptual QA benchmark, featuring a pioneering three-dimensional design principle: “foundational concepts + e-commerce generality + e-commerce specificity.” We propose an integrated, scalable data construction paradigm combining LLM-based automated verification, RAG-enabled consistency checking, and multi-round expert annotation. The benchmark comprises 2,100+ high-quality QA pairs, systematically exposing factual errors, weak generalization, and domain transfer bottlenecks of mainstream Chinese LLMs in e-commerce concept understanding. Experiments demonstrate that ChineseEcomQA significantly improves evaluation reproducibility, attribution fidelity, and domain adaptability—establishing the first standardized, concept-level evaluation tool for e-commerce AI model development and deployment.

Technology Category

• Natural Language Processing: Question Answering
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Search and Optimization: Evaluation and Analysis

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

With the increasing use of Large Language Models (LLMs) in fields such as e-commerce, domain-specific concept evaluation benchmarks are crucial for assessing their domain capabilities. Existing LLMs may generate factually incorrect information within the complex e-commerce applications. Therefore, it is necessary to build an e-commerce concept benchmark. Existing benchmarks encounter two primary challenges: (1) handle the heterogeneous and diverse nature of tasks, (2) distinguish between generality and specificity within the e-commerce field. To address these problems, we propose extbf{ChineseEcomQA}, a scalable question-answering benchmark focused on fundamental e-commerce concepts. ChineseEcomQA is built on three core characteristics: extbf{Focus on Fundamental Concept}, extbf{E-commerce Generality} and extbf{E-commerce Expertise}. Fundamental concepts are designed to be applicable across a diverse array of e-commerce tasks, thus addressing the challenge of heterogeneity and diversity. Additionally, by carefully balancing generality and specificity, ChineseEcomQA effectively differentiates between broad e-commerce concepts, allowing for precise validation of domain capabilities. We achieve this through a scalable benchmark construction process that combines LLM validation, Retrieval-Augmented Generation (RAG) validation, and rigorous manual annotation. Based on ChineseEcomQA, we conduct extensive evaluations on mainstream LLMs and provide some valuable insights. We hope that ChineseEcomQA could guide future domain-specific evaluations, and facilitate broader LLM adoption in e-commerce applications.

Problem

• Evaluate LLMs in e-commerce domain
• Address heterogeneous e-commerce tasks
• Balance e-commerce generality and specificity

Innovation

• Scalable e-commerce QA benchmark
• LLM and RAG validation
• Balanced generality and specificity