Omni-LiveAvatar: Minute-Level Real-Time Streaming Joint Audio-Visual Avatar Generation

2026-08-07

AI Summary

This study addresses the bottleneck of existing audio-video generation models in achieving minute-level real-time streaming interaction by proposing the first framework capable of such performance for digital human generation. Through progressive autoregressive distillation, synchronized audio-video LSTMs, and hierarchical rolling prompt planning, the method enables high-quality, long-horizon joint generation with low latency without auxiliary stabilization mechanisms. Experiments demonstrate that the framework achieves a 33× speedup over the teacher model on H200 GPUs, supporting minute-scale real-time inference. Furthermore, it significantly outperforms current acceleration baselines in visual-audio quality and cross-modal synchronization, effectively breaking through the performance limitations of real-time long-video generation.

Abstract

Joint audio-video generative models serve as foundation for immersive and interactive digital-human generation. Nevertheless, most existing models rely on bidirectional attention and multi-step denoising and can generate only short clips, making them unsuitable for real-time interaction over extended durations. We present Omni-LiveAvatar, the first framework for minute-level, real-time streaming joint audio-video avatar generation. Specifically, we propose (1) a progressive autoregressive distillation pipeline that transfers a large bidirectional joint audio-video diffusion model into a few-step autoregressive generator without auxiliary stabilization mechanisms; (2) a synchronized audio-video long-short-term memory that preserves global consistency under a bounded memory budget; and (3) a hierarchical rolling prompt planning strategy that enables coherent semantic evolution and seamless prompt transitions. Extensive experiments show that Omni-LiveAvatar generates high-quality, synchronized minute-level avatars in real time. In terms of speed, it achieves a 33$\times$ generation speedup over its teacher, LTX-2, on a single NVIDIA H200 GPU; in terms of generation quality, it outperforms accelerated baselines across visual quality, audio quality, cross-modal synchronization, and human fidelity. Our code is available at https://github.com/Aoko955/Omni-LiveAvatar.

Problem

• Real-time streaming
• Audio-visual avatar generation
• Long-duration synthesis
• Digital human interaction

Innovation

• Real-time Streaming Avatar Generation
• Progressive Autoregressive Distillation
• Synchronized Audio-Video LSTM
• Hierarchical Rolling Prompt Planning
• Joint Audio-Visual Diffusion