ConvMix: A Mixed-Criteria Data Augmentation Framework for Conversational Dense Retrieval

2025-08-05

AI Summary

To address the scarcity of context-aware query-document relevance annotations in conversational dense retrieval, this paper proposes ConvMix—a large language model–based framework featuring a bidirectional augmentation strategy (query generation and document rewriting), integrated with a hybrid judgment criterion and semantic diversity control. It further incorporates approximate distribution supervision and multi-level quality filtering to ensure both reliability and distributional consistency of synthetic data. Compared to conventional data augmentation approaches, ConvMix significantly improves coverage breadth and scenario adaptability. Extensive experiments across five mainstream benchmarks demonstrate that retrieval models trained with ConvMix consistently outperform state-of-the-art baselines, validating substantial advances in data efficiency and generalization capability.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Conversational search aims to satisfy users' complex information needs via multiple-turn interactions. The key challenge lies in revealing real users' search intent from the context-dependent queries. Previous studies achieve conversational search by fine-tuning a conversational dense retriever with relevance judgments between pairs of context-dependent queries and documents. However, this training paradigm encounters data scarcity issues. To this end, we propose ConvMix, a mixed-criteria framework to augment conversational dense retrieval, which covers more aspects than existing data augmentation frameworks. We design a two-sided relevance judgment augmentation schema in a scalable manner via the aid of large language models. Besides, we integrate the framework with quality control mechanisms to obtain semantically diverse samples and near-distribution supervisions to combine various annotated data. Experimental results on five widely used benchmarks show that the conversational dense retriever trained by our ConvMix framework outperforms previous baseline methods, which demonstrates our superior effectiveness.

Problem

• Addresses data scarcity in conversational dense retrieval training
• Enhances query-document relevance judgment via LLM-augmented framework
• Improves retrieval performance across multiple conversational search benchmarks

Innovation

• Mixed-criteria framework for data augmentation
• Two-sided relevance judgment augmentation using LLMs
• Quality control for diverse semantic samples