CoDeTT: A Context-Aware Decision Benchmark for Turn-Taking Evaluation

2026-03-26

AI Summary

This work addresses the limitations of current turn-taking modeling evaluation in spoken dialogue systems, which is often confined to binary boundary detection and lacks systematic support for complex interactive scenarios. The study reframes turn-taking modeling as a structured decision-making problem and introduces the first context-aware evaluation framework. It establishes a standardized benchmark dataset encompassing diverse interaction scenarios and fine-grained decision categories, along with controlled contextual variations and a unified evaluation protocol. Experimental results reveal significant performance disparities among existing models across different decision types and interaction contexts. The publicly released, reproducible dataset and accompanying toolkit substantially advance the standardization and in-depth analysis of turn-taking system evaluation.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Evaluation and Analysis
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Turn-taking modeling is fundamental to spoken dialogue systems, yet its evaluation remains fragmented and often limited to binary boundary detection under narrow interaction settings. Such protocols hinder systematic comparison and obscure model weaknesses across conversational conditions. We present CoDeTT, a context-aware decision benchmark for turn-taking evaluation. CoDeTT formulates turn-taking as a structured decision problem and constructs a multi-scenario dataset with fine-grained decision categories and controlled context variations. Under a unified evaluation protocol, we assess representative existing models and observe substantial performance disparities across decision types and interaction scenarios. CoDeTT provides a standardized benchmark for systematic and context-aware evaluation of turn-taking systems. The benchmark dataset and evaluation toolkit are available at https://github.com/YingaoWang-casia/CoDeTT.github.io.

Problem

• turn-taking
• evaluation benchmark
• spoken dialogue systems
• context-aware
• decision modeling

Innovation

• turn-taking
• context-aware evaluation
• structured decision
• benchmark dataset
• spoken dialogue systems