Real-Time Voice AI Hears but Does Not Listen

2026-06-24

AI Summary

Current real-time speech AI systems commonly disregard prosodic and emotional acoustic cues, relying predominantly on textual content for decision-making—a limitation that can lead to critical misjudgments in high-stakes scenarios. This work introduces the concept of the “emotional intelligence gap” in speech AI, highlighting a critical disconnect: while these systems can detect affective states, they fail to incorporate such information into downstream decisions. Using a black-box evaluation framework, we assess four leading commercial systems—OpenAI, Google, Alibaba, and another major provider—on multimodal spoken tasks involving crying, fear, sarcasm, and other emotionally charged expressions. Our findings demonstrate that all evaluated systems base their judgments almost exclusively on lexical content rather than vocal characteristics, revealing a significant shortcoming in the integration of emotional intelligence into practical speech AI applications.

Abstract

Speech conveys information through both words and vocal delivery. We evaluate four leading production realtime voice systems-OpenAI's GPT Realtime 2, Google's Gemini 3.1 Flash Live, and Alibaba's Qwen3.5 Omni Plus and Omni Flash-on tasks where the words and the delivery patterns both convey meaningful information. Across three consequential scenarios, all four systems act on the words rather than the voice. They end calls with crying callers who insist nothing is wrong, approve wire transfers authorized in frightened voices, and enroll callers whose agreement is clearly sarcastic. Surprisingly, this is often not a failure of perception. When asked directly, three of the four systems reliably identify the distress, fear, or sarcasm they later ignore when making decisions. We observe a similar pattern when these realtime voice systems estimate accent and age, as their responses frequently follow the biases of the words rather than the acoustic properties of the speaker. We term this disconnect between perception and action the emotional intelligence gap of voice AI. Prompting systems to explicitly attend to vocal delivery improves performance only partially and inconsistently. Our findings show that current realtime voice AI systems often behave as if speech had been reduced to a transcript, suggesting that they should be used with caution in settings where the tone and emotion of delivery convey important information.

Problem

• real-time voice AI
• emotional intelligence gap
• vocal delivery
• speech perception
• voice emotion

Innovation

• emotional intelligence gap
• real-time voice AI
• vocal delivery
• speech perception vs. action
• affective computing