Human or Machine? A Preliminary Turing Test for Speech-to-Speech Interaction

2026-02-27

AI Summary

This work addresses the lack of systematic evaluation of humanness in current speech-to-speech (S2S) systems by proposing the first Turing test framework tailored for S2S. Drawing on nearly 3,000 human judgments, the study establishes a fine-grained humanness taxonomy spanning 18 dimensions and develops an interpretable discriminative model capable of automatically distinguishing human–machine interactions. Experimental results demonstrate that none of the evaluated S2S systems pass the proposed Turing test, with key limitations rooted in paralinguistic cues, emotional expressiveness, and conversational persona. The introduced model achieves high accuracy while maintaining transparency in human–machine classification, offering actionable diagnostic insights to guide future system improvements.

Technology Category

• Natural Language Processing: Speech
• Humans and AI: Human-in-the-loop Machine Learning
• Machine Learning: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

The pursuit of human-like conversational agents has long been guided by the Turing test. For modern speech-to-speech (S2S) systems, a critical yet unanswered question is whether they can converse like humans. To tackle this, we conduct the first Turing test for S2S systems, collecting 2,968 human judgments on dialogues between 9 state-of-the-art S2S systems and 28 human participants. Our results deliver a clear finding: no existing evaluated S2S system passes the test, revealing a significant gap in human-likeness. To diagnose this failure, we develop a fine-grained taxonomy of 18 human-likeness dimensions and crowd-annotate our collected dialogues accordingly. Our analysis shows that the bottleneck is not semantic understanding but stems from paralinguistic features, emotional expressivity, and conversational persona. Furthermore, we find that off-the-shelf AI models perform unreliably as Turing test judges. In response, we propose an interpretable model that leverages the fine-grained human-likeness ratings and delivers accurate and transparent human-vs-machine discrimination, offering a powerful tool for automatic human-likeness evaluation. Our work establishes the first human-likeness evaluation for S2S systems and moves beyond binary outcomes to enable detailed diagnostic insights, paving the way for human-like improvements in conversational AI systems.

Problem

• speech-to-speech
• Turing test
• human-likeness
• conversational AI
• paralinguistic features

Innovation

• speech-to-speech systems
• Turing test
• human-likeness evaluation
• paralinguistic features
• interpretable model