From Speech to Interaction: Analyzing Multimodal Systems in Cocktail-Party Scenarios

2026-08-09

AI Summary

This work addresses the challenge of overlapping speech in cocktail-party scenarios, where conventional automatic speech recognition systems struggle to accurately separate and transcribe individual speakers. The authors propose a multimodal approach that integrates audiovisual target speech separation, multimodal speech recognition, and large language models to jointly perform speaker grouping and conversational modeling. Their method demonstrates the complementary roles of audiovisual cues and linguistic priors in enhancing consistency between speech separation and recognition, thereby challenging the long-held assumption that speech overlap alone constitutes the primary obstacle. Experimental results show that the proposed system achieves up to a 57% relative reduction in word error rate on complex multi-speaker dialogues, significantly outperforming existing state-of-the-art methods.

Abstract

Humans have the remarkable ability to engage in spontaneous informal conversations and selectively attend to individual speakers while filtering out competing speech from nearby conversations. This "cocktail party" scenario still presents severe challenges to speech recognition systems. The CHiME-9 MCoRec task provides a testbed where systems must recognize groups of speakers and transcribe each of their conversations from audio-visual input. In this work, we analyze a diverse set of systems, representing different design directions for addressing the cocktail-party scenario, where the best system achieves up to 57% relative error reduction. We identify three main strategies: (1) explicit or implicit audio-visual target speech separation, (2) improved audio-visual speech recognition for each target speaker, and (3) the use of large language models to group speakers into conversations and enhance conversational consistency. Our analysis shows that these directions address complementary failure modes of the cocktail-party problem, and that high speech overlap alone does not explain performance differences, challenging the common assumption that overlap is the primary source of difficulty in cocktail-party recognition.

Problem

• cocktail-party problem
• speech recognition
• multimodal systems
• speaker separation
• audio-visual input

Innovation

• audio-visual speech separation
• multimodal speech recognition
• large language models
• cocktail-party problem
• speaker diarization