DeepThink: Aligning Language Models with Domain-Specific User Intents

2025-02-08

AI Summary

In advertising-domain question answering, synthetic instructions often misalign with genuine user intent. To address this, we propose a fully automated, three-stage instruction generation framework: (1) seed-question-driven initial instruction construction; (2) implicit requirement elicitation via multi-turn dialogue simulation; and (3) answer rewriting and refinement by jointly leveraging RAG-retrieved documents and dialogue context. Our approach unifies instruction synthesis, dialogue modeling, retrieval-augmented generation, and context-aware response generation. Evaluated on a real-user test set, it outperforms the GPT-4-turbo+RAG baseline across five dimensions—relevance, completeness, clarity, accuracy, and actionability—with an average improvement of 7.92%. This demonstrates substantial gains in modeling authentic user intent and enhancing response quality.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Intelligent User Interfaces
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Supervised fine-tuning with synthesized instructions has been a common practice for adapting LLMs to domain-specific QA tasks. However, the synthesized instructions deviate from real user questions and expected answers. This study proposes a novel framework called DeepThink to generate high-quality instructions. DeepThink first generates a few seed questions to mimic actual user questions, simulates conversations to uncover the hidden user needs, and refines the answer by conversational contexts and the retrieved documents for more comprehensive answers. Experiments demonstrate that DeepThink achieves an average performance improvement of 7.92% compared to a GPT-4-turbo+RAG-based assistant on the real user test set in the advertising domain across dimensions such as relevance, completeness, clarity, accuracy, and actionability.

Problem

• Improves domain-specific QA accuracy
• Generates realistic user questions
• Enhances answer relevance and completeness

Innovation

• Generates seed questions mimicking user queries
• Simulates conversations to uncover hidden needs
• Refines answers using conversational contexts and documents