Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interaction

2026-05-17

AI Summary

This work addresses the lack of systematic evaluation for real-time duplex multimodal interaction in existing methods. It introduces the first comprehensive benchmark tailored to this setting, encompassing two task types—real-time description and proactive alerting—across nine real-world scenarios with 660 videos annotated with fine-grained temporal labels. The study further proposes an LLM-as-a-Judge automated evaluation framework enhanced with time-awareness and sequential reasoning mechanisms to jointly assess both the content and timing of system responses. Experimental results reveal that even state-of-the-art models achieve only a 39.6% overall score, with performance on proactive alerting tasks dropping as low as 20.0%, highlighting significant challenges in jointly modeling response timing and coherent generation.

Abstract

Real-time duplex interaction is essential for multimodal AI systems operating in real-world scenarios, where models must continuously process streaming inputs and respond at appropriate moments. However, most existing multimodal large language models (MLLMs) are evaluated in offline settings, where the entire video input is processed before any response is generated. While recent work has started to explore real-time duplex MLLMs, there is still no comprehensive benchmark or automatic evaluation method for this setting. To address this gap, we propose Omni-DuplexEval, a benchmark for systematically evaluating real-time duplex interaction. The benchmark consists of two complementary scenarios: (1) Real-Time Description, which evaluates the ability to generate continuous, time-aligned responses that track evolving multimodal inputs, and (2) Proactive Reminder, which evaluates the ability to identify salient events and respond at appropriate moments. Omni-DuplexEval contains 660 videos with fine-grained, human-annotated labels and precise temporal metadata, spanning 9 tasks grounded in real-world scenarios, where all questions are formulated as open-ended queries. We further introduce an automatic evaluation framework based on LLM-as-a-Judge, which enables systematic assessment by jointly evaluating response-content alignment and response timing through timestamp-aware and sequential reasoning, achieving strong alignment with human judgments. Experiments on state-of-the-art duplex MLLMs reveal substantial limitations. The best-performing model achieves only 39.6% overall, while scoring only 20.0% on Proactive Reminder. Our analysis identifies two key challenges: models struggle to balance timely responses with coherent, holistic content generation, and they often fail to determine both when to respond and what to produce. We hope our work facilitates further progress in MLLMs.

Problem

• real-time duplex interaction
• multimodal large language models
• evaluation benchmark
• streaming inputs
• response timing

Innovation

• real-time duplex interaction
• multimodal large language models
• temporal alignment
• proactive response
• automatic evaluation