EchoTorrent: Towards Swift, Sustained, and Streaming Multi-Modal Video Generation

2026-02-14

AI Summary

Existing streaming multimodal video generation models suffer from high latency, temporal instability, spatial blurriness, temporal drift, and lip-audio asynchrony, hindering their applicability in real-time scenarios. This work proposes EchoTorrent, a novel framework that achieves efficient and stable few-step autoregressive generation through four synergistic innovations: multi-teacher knowledge distillation, adaptive classifier-free guidance calibration (ACC-DMD), forced alignment of long-tail frames within a causal–bidirectional hybrid architecture, and optimization of the pixel-domain VAE decoder. EchoTorrent substantially enhances temporal consistency, identity preservation, and lip-sync accuracy, effectively mitigating multimodal degradation inherent in streaming generation and striking a breakthrough balance between efficiency and generation quality.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Recent multi-modal video generation models have achieved high visual quality, but their prohibitive latency and limited temporal stability hinder real-time deployment. Streaming inference exacerbates these issues, leading to pronounced multimodal degradation, such as spatial blurring, temporal drift, and lip desynchronization, which creates an unresolved efficiency-performance trade-off. To this end, we propose EchoTorrent, a novel schema with a fourfold design: (1) Multi-Teacher Training fine-tunes a pre-trained model on distinct preference domains to obtain specialized domain experts, which sequentially transfer domain-specific knowledge to a student model; (2) Adaptive CFG Calibration (ACC-DMD), which calibrates the audio CFG augmentation errors in DMD via a phased spatiotemporal schedule, eliminating redundant CFG computations and enabling single-pass inference per step; (3) Hybrid Long Tail Forcing, which enforces alignment exclusively on tail frames during long-horizon self-rollout training via a causal-bidirectional hybrid architecture, effectively mitigates spatiotemporal degradation in streaming mode while enhancing fidelity to reference frames; and (4) VAE Decoder Refiner through pixel-domain optimization of the VAE decoder to recover high-frequency details while circumventing latent-space ambiguities. Extensive experiments and analysis demonstrate that EchoTorrent achieves few-pass autoregressive generation with substantially extended temporal consistency, identity preservation, and audio-lip synchronization.

Problem

• multi-modal video generation
• streaming inference
• temporal stability
• audio-lip synchronization
• real-time deployment

Innovation

• Streaming Video Generation
• Multi-Teacher Knowledge Transfer
• Adaptive CFG Calibration
• Hybrid Long Tail Forcing
• VAE Decoder Refinement