Building Open-Retrieval Conversational Question Answering Systems by Generating Synthetic Data and Decontextualizing User Questions

2025-07-07

AI Summary

Open-retrieval conversational question answering (OR-CONVQA) suffers from scarce domain-specific annotated data, hindering system development. Method: This paper proposes an end-to-end, annotation-free framework: (1) an automated pipeline for generating high-fidelity, multi-turn synthetic dialogue data; (2) joint modeling of question decontextualization and rewriting to decouple user queries from dialogue history, enabling direct reuse of off-the-shelf general-purpose retrievers; and (3) an LLM-based, interpretable response generation module that integrates retrieved evidence with dialogue state. Contribution/Results: Experiments demonstrate substantial improvements in question rewriting quality and retrieval accuracy across multiple domains. The approach achieves high-quality, low-dependency conversational QA under data-scarce conditions, establishing a scalable technical paradigm for OR-CONVQA in annotation-limited settings.

Technology Category

• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web question answering
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

We consider open-retrieval conversational question answering (OR-CONVQA), an extension of question answering where system responses need to be (i) aware of dialog history and (ii) grounded in documents (or document fragments) retrieved per question. Domain-specific OR-CONVQA training datasets are crucial for real-world applications, but hard to obtain. We propose a pipeline that capitalizes on the abundance of plain text documents in organizations (e.g., product documentation) to automatically produce realistic OR-CONVQA dialogs with annotations. Similarly to real-world humanannotated OR-CONVQA datasets, we generate in-dialog question-answer pairs, self-contained (decontextualized, e.g., no referring expressions) versions of user questions, and propositions (sentences expressing prominent information from the documents) the system responses are grounded in. We show how the synthetic dialogs can be used to train efficient question rewriters that decontextualize user questions, allowing existing dialog-unaware retrievers to be utilized. The retrieved information and the decontextualized question are then passed on to an LLM that generates the system's response.

Problem

• Generating synthetic data for OR-CONVQA training
• Decontextualizing user questions for better retrieval
• Training efficient question rewriters using synthetic dialogs

Innovation

• Generating synthetic OR-CONVQA dialogs automatically
• Decontextualizing user questions for retrieval compatibility
• Leveraging LLMs for grounded system responses