M3-SLU: Evaluating Speaker-Attributed Reasoning in Multimodal Large Language Models

2025-10-22

AI Summary

This work addresses the weak speaker attribution capability of multimodal large language models (MLLMs) in multi-speaker, multi-turn dialogues. We introduce the first dedicated benchmark for evaluating “who said what and when” in multimodal settings. The benchmark integrates four corpora—CHiME-6, MELD, MultiDialog, and AMI—providing synchronized audio, transcribed text, and speaker metadata to support contextualized, fine-grained understanding. We propose a novel dual-task evaluation paradigm—question answering and utterance matching—assessed automatically via LLM-as-Judge and accuracy metrics. Experiments on over 12,000 samples reveal that while current MLLMs demonstrate strong comprehension of utterance content, their speaker attribution accuracy remains significantly low. Our benchmark establishes a high-challenge, reproducible evaluation framework and provides systematic analysis to advance speaker-aware multimodal dialogue understanding.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

We present M3-SLU, a new multimodal large language model (MLLM) benchmark for evaluating multi-speaker, multi-turn spoken language understanding. While recent models show strong performance in speech and text comprehension, they still struggle with speaker-attributed reasoning, the ability to understand who said what and when in natural conversations. M3-SLU is built from four open corpora (CHiME-6, MELD, MultiDialog, and AMI) and comprises over 12,000 validated instances with paired audio, transcripts, and metadata. It includes two tasks: (1) Speaker-Attributed Question Answering and (2) Speaker Attribution via Utterance Matching. We provide baseline results for both cascaded pipelines and end-to-end MLLMs, evaluated using an LLM-as-Judge and accuracy metrics. Results show that while models can capture what was said, they often fail to identify who said it, revealing a key gap in speaker-aware dialogue understanding. M3-SLU offers as a challenging benchmark to advance research in speaker-aware multimodal understanding.

Problem

• Evaluating speaker-attributed reasoning in multimodal language models
• Assessing who said what and when in multi-turn conversations
• Addressing failures in identifying speakers despite understanding content

Innovation

• M3-SLU benchmark evaluates speaker-attributed multimodal reasoning
• Combines four corpora with audio transcripts and metadata
• Tests speaker identification via question answering and utterance matching