TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialogue

2026-07-01

AI Summary

This work addresses the lack of a unified, automated framework for cross-type evaluation of naturalness in dyadic turn-taking, which has traditionally relied on subjective human judgments or narrow temporal metrics. The authors propose TurnNat, a novel framework that leverages a causal voice activity prediction model to quantify temporal anomalies via negative log-likelihood (NLL) and introduces Turn Boundary Units (TBUs) to aggregate frame-level scores into dialogue-level naturalness ratings. They construct the first human-validated benchmark dataset with controlled perturbations and employ both mean and tail-aggregation strategies for scoring. Experimental results demonstrate that TurnNat effectively detects diverse heterogeneous temporal anomalies and yields naturalness scores that align closely with human judgments, significantly outperforming existing automatic evaluation methods.

Abstract

Turn-taking naturalness is central to full-duplex spoken dialogue systems, yet its automatic evaluation remains limited. Existing evaluations often rely on human judgments or behavior-specific timing metrics, making it difficult to compare heterogeneous timing failures within a unified framework. We propose TurnNat, a likelihood-based framework for automatic turn-taking naturalness evaluation in two-channel spoken dialogue. A causal turn-taking prediction model trained on natural conversations estimates future two-speaker voice-activity states, and the negative log-likelihood (NLL) of the observed future activity measures timing atypicality. TurnNat pools frame-level NLLs over turn-taking boundary units (TBUs) extracted from utterance onsets and offsets, and aggregates mean and tail TBU scores into a dialogue-level naturalness score. We further construct a controlled perturbation benchmark of paired natural and perturbed dialogue clips, validated by human naturalness judgments. Experiments on this benchmark show that TurnNat successfully identifies unnatural turn-taking perturbations across heterogeneous timing failures.

Problem

• turn-taking naturalness
• spoken dialogue
• automatic evaluation
• timing failures
• full-duplex dialogue

Innovation

• turn-taking naturalness
• automatic evaluation
• negative log-likelihood
• causal prediction model
• turn-taking boundary units