Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence

2026-08-11

AI Summary

This work addresses the lack of coordinated visual expression in existing full-modal dialogue systems, which often generate speech responses without semantically aligned visual output, resulting in audio-visual misalignment. To bridge this gap, the authors propose a unified framework that jointly generates semantically consistent text, personalized speech, and conditionally synthesized video from multimodal queries, reference images, and audio inputs. Key innovations include a Visual Thought Planning (VTP) module for structured modeling of scenes, emotions, and actions; a shared multi-codebook acoustic unit serving as a unified interface to align speech and video generation; and a block-causal streaming student model integrated with a prefix-streaming mechanism to enable efficient incremental synthesis. Implemented in an end-to-end four-GPU pipeline, the system achieves 1.293× real-time inference at 400×720 or 720×400 resolution, striking a practical balance between generation quality and computational efficiency.

Abstract

Omni-modal dialogue models can understand multimodal inputs and synthesize spoken replies, yet their responses remain visually disembodied. We introduce \textbf{Ex-Omni-2D}, an omni-modal dialogue framework that generates a coordinated response comprising text, personalized speech, and reference-conditioned video. Given a multimodal query, reference image, and reference audio, the model predicts a structured \textit{Visual Thought Plan} (VTP) describing scene, emotion, and motion, followed by response text and native multi-codebook speech units. These units form a shared acoustic-temporal interface: they are decoded into speech and aligned online with video frames. This interface enables the response and avatar pathways to be learned from heterogeneous speech, dialogue, and avatar-video data, avoiding the need for large-scale query--text--speech--video supervision. A full-sequence Video Generator serves as the primary Teacher. For efficient incremental generation, we further distill it into a few-step block-causal \emph{Streaming Student} whose Prefix Streaming mechanism carries a clean latent across consecutive chunks to reduce cumulative late-chunk degradation. With four-step inference, the complete four-GPU pipeline achieves an end-to-end RTF of 1.293 at $400\times720$/$720\times400$, providing a practical quality--efficiency operating point.

Problem

• omni-modal dialogue
• visually disembodied response
• coordinated audio-visual generation
• avatar-video synthesis
• multimodal interaction

Innovation

• omni-modal dialogue
• Visual Thought Plan
• multi-codebook speech units
• streaming video generation
• acoustic-temporal interface