🤖 AI Summary
Existing dialogue benchmarks inadequately assess the persuasive capability and conversion efficacy of large language models (LLMs) in sales scenarios. To address this gap, this work proposes SalesLLM, the first bilingual sales dialogue benchmark explicitly centered on transaction outcomes, spanning financial and consumer goods domains with over 30,000 script configurations and more than 1,800 multi-turn dialogues, featuring controllable difficulty levels and role specifications. We develop an end-to-end automated evaluation pipeline integrating an LLM-based scorer and a fine-tuned BERT classifier, alongside a trained CustomerLM user simulator to mitigate role confusion. Experiments demonstrate that SalesLLM scores exhibit strong alignment with expert human judgments (Pearson r = 0.98), effectively differentiate performance across 15 mainstream LLMs, and reveal that certain models achieve human-level proficiency, thereby validating the benchmark’s validity and scalability.
📝 Abstract
Sales dialogues require multi-turn, goal-directed persuasion under asymmetric incentives, which makes them a challenging setting for large language models (LLMs). Yet existing dialogue benchmarks rarely measure deal progression and outcomes. We introduce SalesLLM, a bilingual (ZH/EN) benchmark derived from realistic applications covering Financial Services and Consumer Goods, built from 30,074 scripted configurations and 1,805 curated multi-turn scenarios with controllable difficulty and personas. We propose a fully automatic evaluation pipeline that combines (i) an LLM-based rater for sales-process progress, and (ii) fine-tuned BERT classifiers for end-of-dialogue buying intent. To improve simulation fidelity, we train a user model, CustomerLM, with SFT and DPO on 8,000 crowdworker-involved sales conversations, reducing role inversion from 17.44% (GPT-4o) to 8.8%. SalesLLM scores correlate strongly with expert human ratings (Pearson r=0.98). Experiments across 15 mainstream LLMs reveal substantial variability: top-performance LLMs are competitive with human-level performance while the less capable ones are worse than human. SalesLLM serves as a scalable benchmark for developing and evaluating outcome-oriented sales agents.