Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models

2026-06-23

AI Summary

This work addresses the high latency and error accumulation inherent in traditional cascaded systems for real-time audiovisual interaction, which stem from modular fragmentation. To overcome these limitations, the authors propose the first end-to-end native streaming multimodal foundation model that unifies language, audio, and video into an interleaved token sequence processed within a single Transformer architecture, enabling full-duplex interaction. The system employs a causal encoder–decoder, block-wise causal attention, and a low-latency multimodal token scheduling strategy to support incremental processing at 160 ms granularity (25 fps) without relying on external speech recognition, synthesis, or animation modules. Empirical evaluation demonstrates an on-device response latency of approximately 200 ms; combined with 350 ms network latency, the total interaction latency reaches about 550 ms, marking the first achievement of sub-second full-duplex audiovisual communication.

Abstract

We present Wan-Streamer, a native-streaming, end-to-end interactive foundation model designed from the ground up for real-time, low-latency, full-duplex audio-visual interaction. Wan-Streamer seamlessly models language, audio, and video as both input and output within a single Transformer, where the sequence is represented as interleaved visual, audio, and text input tokens together with visual, audio, and text output tokens, coordinated by block-causal attention for incremental streaming. Unlike cascaded interactive systems that rely on separate VAD, ASR, language, TTS, audio-driven animation, or video-generation modules, Wan-Streamer does not rely on external language, speech, avatar, or video-generation modules: perception, reasoning, generation, response timing, turn management, and cross-modal synchronization are learned jointly within one unified model, reducing pipeline latency and error accumulation. To support natural audio-visual responsiveness, we redesign the entire stack around streamability, including causal encoders, causal decoders, block-causal attention, and low-latency multimodal token scheduling, enabling streaming units as short as 160 ms at 25 fps. Wan-Streamer achieves approximately 200 ms model-side response latency and approximately 550 ms total interaction latency when combined with 350 ms bidirectional network latency, supporting sub-second duplex audio-visual communication. These results position Wan-Streamer as a unified, end-to-end, multimodal interactive foundation model for low-latency streaming interaction.

Problem

• real-time interaction
• low-latency
• multimodal foundation model
• full-duplex audio-visual communication
• end-to-end streaming

Innovation

• streaming foundation model
• block-causal attention
• end-to-end multimodal interaction
• low-latency audio-visual communication
• unified transformer architecture