Evaluating Large Language Models Abilities for Addressee, Turn-change, and Next Speaker Prediction in Meetings

2026-06-16

AI Summary

This study investigates speaker turn-taking mechanisms in multimodal multiparty dialogue, focusing on addressee identification, turn transition prediction, and next-speaker prediction. Leveraging the AMI meeting corpus, it presents the first systematic evaluation of large language models (LLMs) that have not undergone domain-specific fine-tuning and operate solely on textual input, assessing their ability to predict conversational turn structures. The performance of these text-only LLMs is compared against supervised models, multimodal LLMs, and human annotators. Results show that text-only LLMs outperform both supervised models and humans in next-speaker prediction, while multimodal LLMs surpass text-only counterparts in addressee identification and turn transition prediction, yet still fall short of human performance. This work reveals the implicit capacity of LLMs to model dialogue structure and highlights key similarities and differences between model-based and human predictions.

Abstract

We investigate turn-taking in multimodal multi-party conversations using large language models (LLMs). We construct an evaluation framework for three tasks: addressee detection, turn-change prediction, and next speaker prediction. We compare supervised models trained for these tasks, text-based LLMs, multimodal LLMs (MM-LLMs), and human subjects. Experiments on the AMI corpus showed that LLMs outperformed supervised models and humans in next speaker prediction, despite not being trained on the target domain and without access to audio or visual information. An MM-LLM performed better than text-based LLMs on addressee detection and turn-change prediction but remained below human performance, indicating difficulty leveraging raw audio-visual signals. Ablation analyses revealed that conversational context was critical, particularly for next speaker prediction. We observed that human and LLM prediction patterns were similar, and intervals with frequent turn changes were difficult for both.

Problem

• addressee detection
• turn-change prediction
• next speaker prediction
• large language models
• multimodal conversations

Innovation

• large language models
• turn-taking
• multimodal conversation
• next speaker prediction
• addressee detection