TurnWise: The Gap between Single- and Multi-turn Language Model Capabilities

2026-03-17

AI Summary

Current language models are predominantly trained and evaluated on single-turn dialogues, which inadequately capture or enhance their multi-turn conversational capabilities. To address this limitation, this work introduces TurnWiseEval—the first benchmark enabling direct comparability between single-turn and multi-turn performance—and TurnWiseData, a scalable pipeline for synthesizing multi-turn dialogue data. By integrating a post-training fine-tuning strategy based on OLMo-3, the approach achieves a 12% performance gain on TurnWiseEval using only 10,000 synthetically generated multi-turn dialogues, substantially narrowing the gap between single-turn and multi-turn competencies.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Multi-turn conversations are a common and critical mode of language model interaction. However, current open training and evaluation data focus on single-turn settings, failing to capture the additional dimension of these longer interactions. To understand this multi-/single-turn gap, we first introduce a new benchmark, TurnWiseEval, for multi-turn capabilities that is directly comparable to single-turn chat evaluation. Our evaluation isolates multi-turn specific conversational ability through pairwise comparison to equivalent single-turn settings. We additionally introduce our synthetic multi-turn data pipeline TurnWiseData which allows the scalable generation of multi-turn training data. Our experiments with Olmo 3 show that training with multi-turn data is vital to achieving strong multi-turn chat performance, and that including as little as 10k multi-turn conversations during post-training can lead to a 12% improvement on TurnWiseEval.

Problem

• multi-turn conversation
• single-turn evaluation
• language model capability gap
• conversational AI
• dialogue systems

Innovation

• multi-turn dialogue
• benchmarking
• synthetic data generation
• language model evaluation
• post-training