MTAVG-Bench: A Comprehensive Benchmark for Evaluating Multi-Talker Dialogue-Centric Audio-Video Generation

2026-01-31

AI Summary

Existing evaluation benchmarks struggle to effectively assess critical issues in multi-speaker conversational video generation, such as identity drift, unnatural turn-taking, and audio-visual asynchrony. This work proposes the first fine-grained audiovisual generation evaluation framework tailored to this scenario, introducing a comprehensive benchmark comprising 1.8K videos and 2.4K structured question-answer pairs, constructed via a semi-automatic pipeline. The framework evaluates models across four dimensions: audiovisual fidelity, temporal consistency, social interaction coherence, and cinematic expressiveness, enabling precise failure analysis and targeted model refinement. Experiments on twelve leading open- and closed-source models reveal that Gemini 3 Pro achieves the best overall performance, while certain open-source models excel in signal fidelity and temporal consistency.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Recent advances in text-to-audio-video (T2AV) generation have enabled models to synthesize audio-visual videos with multi-participant dialogues. However, existing evaluation benchmarks remain largely designed for human-recorded videos or single-speaker settings. As a result, potential errors that occur in generated multi-talker dialogue videos, such as identity drift, unnatural turn transitions, and audio-visual misalignment, cannot be effectively captured and analyzed. To address this issue, we introduce MTAVG-Bench, a benchmark for evaluating audio-visual multi-speaker dialogue generation. MTAVG-Bench is built via a semi-automatic pipeline, where 1.8k videos are generated using multiple popular models with carefully designed prompts, yielding 2.4k manually annotated QA pairs. The benchmark evaluates multi-speaker dialogue generation at four levels: audio-visual signal fidelity, temporal attribute consistency, social interaction, and cinematic expression. We benchmark 12 proprietary and open-source omni-models on MTAVG-Bench, with Gemini 3 Pro achieving the strongest overall performance, while leading open-source models remain competitive in signal fidelity and consistency. Overall, MTAVG-Bench enables fine-grained failure analysis for rigorous model comparison and targeted video generation refinement.

Problem

• multi-talker dialogue
• audio-visual generation
• evaluation benchmark
• identity drift
• audio-visual misalignment

Innovation

• multi-talker dialogue
• audio-visual generation
• evaluation benchmark
• temporal consistency
• social interaction modeling