MotionStream: Real-Time Video Generation with Interactive Motion Controls

2025-11-03

AI Summary

Existing motion-conditioned video generation methods suffer from high latency (on the order of minutes) and non-causal inference, rendering them unsuitable for real-time interactive applications. This paper introduces the first millisecond-latency, infinite-duration streaming video generation framework, achieving 29 FPS high-quality output on a single GPU. Our approach addresses the core challenges through three key contributions: (1) a sliding-window causal attention mechanism with attention anchors to enforce strict causality; (2) self-enforced distribution-matching distillation, enabling efficient transfer from non-causal pretrained models to causal inference; and (3) integration of text-video priors, rolling KV caching, self-rollout, and context-aware training. Quantitative and qualitative evaluations demonstrate state-of-the-art performance in motion-following accuracy and visual quality, with generation speed improved by two orders of magnitude over prior work—enabling, for the first time, truly real-time, interactive video streaming.

Technology Category

• Computer Vision: Motion & Tracking
• Natural Language Processing: Generation
• Machine Learning: Causal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Current motion-conditioned video generation methods suffer from prohibitive latency (minutes per video) and non-causal processing that prevents real-time interaction. We present MotionStream, enabling sub-second latency with up to 29 FPS streaming generation on a single GPU. Our approach begins by augmenting a text-to-video model with motion control, which generates high-quality videos that adhere to the global text prompt and local motion guidance, but does not perform inference on the fly. As such, we distill this bidirectional teacher into a causal student through Self Forcing with Distribution Matching Distillation, enabling real-time streaming inference. Several key challenges arise when generating videos of long, potentially infinite time-horizons: (1) bridging the domain gap from training on finite length and extrapolating to infinite horizons, (2) sustaining high quality by preventing error accumulation, and (3) maintaining fast inference, without incurring growth in computational cost due to increasing context windows. A key to our approach is introducing carefully designed sliding-window causal attention, combined with attention sinks. By incorporating self-rollout with attention sinks and KV cache rolling during training, we properly simulate inference-time extrapolations with a fixed context window, enabling constant-speed generation of arbitrarily long videos. Our models achieve state-of-the-art results in motion following and video quality while being two orders of magnitude faster, uniquely enabling infinite-length streaming. With MotionStream, users can paint trajectories, control cameras, or transfer motion, and see results unfold in real-time, delivering a truly interactive experience.

Problem

• Real-time video generation with interactive motion controls
• Overcoming high latency in motion-conditioned video generation
• Enabling infinite-length streaming video with constant computational cost

Innovation

• Distills bidirectional teacher into causal student
• Uses sliding-window causal attention with sinks
• Enables constant-speed infinite-length video generation