Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap

2026-05-23

AI Summary

This work addresses the prevalent “dialogue drift” problem in large language models, where performance in multi-turn conversations significantly degrades compared to single-turn settings. To mitigate this issue, the authors propose a view-asymmetric self-distillation framework that leverages the model’s own single-turn responses as teacher signals to guide its multi-turn dialogue learning—eliminating the need for external teacher models. This approach enables efficient knowledge transfer from single-turn to multi-turn scenarios for the first time. Extensive experiments across mainstream architectures—including Llama, Qwen, Phi, and OLMo—demonstrate that the method recovers at least 92% of single-turn performance in multi-turn settings, with certain Llama variants achieving full parity (100%). The results highlight substantial improvements in both effectiveness and efficiency for multi-turn dialogue generation.

Abstract

Large Language Model (LLM) interactions are typically underspecified, with users clarifying all necessary details across multiple conversational turns. Yet recent work shows that LLMs perform far worse in this multi-turn setting than in a single turn with same information being available at once, a phenomenon termed "Lost-in-Conversation." However, bridging this gap effectively remains an open problem. Here we introduce Found in Conversation (FiC), a training framework where a model teaches itself to find and recover its single-turn competence given underspecified multi-turn prompts. We develop View-Asymmetric Self-Distillation, which distills across two views of the same task information--single-turn view for the teacher, multi-turn view for the student--transferring strong single-turn behavior into weak multi-turn behavior. This requires no stronger external teacher, which is unavailable as even frontier LLMs exhibit this gap. Across model families (Llama, Qwen, Phi, and OLMo) and sizes (3B-14B), FiC recovers at least 92% of single-turn performance and reaches 100% on two Llama backbones, yielding more efficient and helpful multi-turn conversations with single-turn capabilities intact.

Problem

• Lost-in-Conversation
• multi-turn dialogue
• Large Language Models
• performance gap
• conversational AI

Innovation

• Found in Conversation
• View-Asymmetric Self-Distillation
• multi-turn dialogue
• self-distillation
• Lost-in-Conversation