Reference-Based Prosody and Rhythm Evaluation for Spoken Dialogue Systems

2026-06-29

AI Summary

Current spoken dialogue systems lack interpretable prosodic evaluation methods that adapt to speaker characteristics and interaction states. This work addresses this gap by proposing a condition-matched human reference benchmark and a percentile-based evaluation protocol, leveraging over 4,000 hours of dyadic English conversational data to enable fine-grained analysis of acoustic prosodic features such as fundamental frequency, speech rate, and pause duration. By incorporating hierarchical matching and out-of-bounds flagging mechanisms, the method significantly enhances behavioral plausibility and interpretability of prosodic assessments. Validation on held-out human data demonstrates an anomaly flagging rate close to the nominal 10%, outperforming conventional aggregate statistics while clearly indicating the direction of deviation, thereby providing an effective behavioral plausibility check for synthetic speech.

Abstract

Speech-to-speech (S2S) AI agents are advancing rapidly, yet evaluation lacks interpretable speech-native measures for conversational prosody and rhythm. Because $F_0$, speaking rate, articulation rate, and pausing shift with model-predicted speaker traits and interaction state, pooled human statistics can be poorly calibrated for evaluating a particular output. Using 4000+ hours of dyadic English conversation from the Seamless Interaction dataset, we construct matched reference regimes for $F_0$ mean, $F_0$ expressivity, speech rate, articulation rate, pause ratio, and mean pause duration. We then define a percentile-based evaluation protocol: extract the same metrics from an S2S output waveform, compare them to the closest matched human reference stratum, and report percentile deviations or 5th-95th percentile out-of-regime flags. On held-out human rows, pooled references over-flag state-conditioned $F_0$ expressivity and rhythm, while matched references return flag rates closer to the nominal 10% and make deviation direction interpretable. These outputs serve as behavioral plausibility checks that complement, rather than replace, perceptual and user-centered evaluation.

Problem

• prosody evaluation
• rhythm assessment
• speech-to-speech systems
• reference-based metrics
• conversational speech

Innovation

• reference-based evaluation
• prosody modeling
• rhythm assessment
• percentile deviation
• matched reference strata