SynthAVE: Scalable Synthetic Labeling for E-Commerce with LLM-Arena Validation

2026-07-08

AI Summary

This work addresses the challenges of large-scale e-commerce attribute extraction, where high-quality annotated data across diverse categories and multiple languages is scarce and manual labeling is prohibitively expensive. The authors propose a synthetic label validation framework leveraging a multi-large language model (LLM) arena combined with a majority voting mechanism. The approach automatically evaluates generated labels using 21 referee configurations—comprising seven model families and three prompt templates—and ensures annotation consistency through Cohen’s κ and Fleiss’ κ metrics. Evaluated on a benchmark encompassing 12,726 products, 229 categories, 792 attributes, and four languages, the synthetic labels demonstrate strong agreement with expert annotations (κ = 0.92, 95.2% concordance), enabling a cost-effective, scalable, and highly accurate labeling pipeline.

Abstract

Fine-tuning large language models (LLMs) for e-commerce attribute extraction requires labeled data representative across thousands of product types, attributes, and multiple languages. This combinatorial scale translates to millions of annotations, rendering human labeling prohibitively costly. While recent work has demonstrated synthetic label generation using LLMs, deploying such approaches at industrial scale requires integrated quality control mechanisms. We present SynthAVE, a large-scale human-validated benchmark for attribute value extraction spanning 12,726 products across 229 product types, 792 attributes, and 4 languages (Spanish, French, Italian, German). To validate synthetic labels at scale, we introduce a multi-LLM arena framework where samples are independently evaluated by 21 judge configurations (7 model families $\times$ 3 prompts), with final labels determined via majority voting. The majority vote ensemble agrees with human experts at Cohen's $κ= 0.92$ (95.2% agreement), while individual judges show substantial inter-model agreement (Fleiss' $κ= 0.76$). This demonstrates that diverse models with varying individual judgments aggregate into highly reliable predictions, enabling cost-effective validation at scale while maintaining quality parity with human review.

Problem

• synthetic labeling
• e-commerce
• attribute extraction
• large language models
• quality control

Innovation

• synthetic labeling
• LLM-arena validation
• attribute extraction
• majority voting
• e-commerce