Still Between Us? Evaluating and Improving Voice Assistant Robustness to Third-Party Interruptions

2026-04-19

AI Summary

Current voice assistants struggle to distinguish between the primary user and third-party interruptions, often relying on semantic shortcuts that overlook critical acoustic cues, leading to contextual misunderstandings. This work addresses this limitation by introducing TPI-Train, the first dataset specifically designed for third-party interruption (TPI) detection, along with TPI-Bench, a corresponding evaluation benchmark. By incorporating speaker-aware hard negative samples, the proposed approach enhances the model’s ability to leverage acoustic signals effectively. Experimental results demonstrate a significant improvement in interruption detection accuracy under multi-speaker interference scenarios, advancing speech-language models toward more robust, multimodal interactive capabilities.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

While recent Spoken Language Models (SLMs) have been actively deployed in real-world scenarios, they lack the capability to discern Third-Party Interruptions (TPI) from the primary user's ongoing flow, leaving them vulnerable to contextual failures. To bridge this gap, we introduce TPI-Train, a dataset of 88K instances designed with speaker-aware hard negatives to enforce acoustic cue prioritization for interruption handling, and TPI-Bench, a comprehensive evaluation framework designed to rigorously measure the interruption-handling strategy and precise speaker discrimination in deceptive contexts. Experiments demonstrate that our dataset design mitigates semantic shortcut learning-a critical pitfall where models exploit semantic context while neglecting acoustic signals essential for discerning speaker changes. We believe our work establishes a foundational resource for overcoming text-dominated unimodal reliance in SLMs, paving the way for more robust multi-party spoken interaction. The code for the framework is publicly available at https://tpi-va.github.io

Problem

• Third-Party Interruptions
• Spoken Language Models
• speaker discrimination
• voice assistant robustness
• acoustic cues

Innovation

• Third-Party Interruptions
• Spoken Language Models
• speaker-aware hard negatives
• acoustic cue prioritization
• semantic shortcut learning