Streaming4D: Accelerate 4D World Models via Block-wise Video Generation and Incremental Reconstruction

2026-08-31

AI Summary

本文提出Streaming4D方法，通过块状视频生成与增量3D重建同步进行的方式，解决4D世界模型生成中的高交互延迟问题。

Abstract

Current 4D generation paradigms are often bottlenecked by a sequential decoupling design: video is generated first, followed by 3D reconstruction, leading to high interaction latency. This limits applications in interactive real-time scenarios. To this end, we propose \textbf{Streaming4D}, a tightly coupled synchronous pipeline that integrates block-wise autoregressive video generation with incremental 3D reconstruction. Unlike traditional frame-by-frame emission and delayed geometry recovery, Streaming4D generates temporal video blocks and immediately triggers reconstruction for each completed block, enabling parallel execution between synthesis and geometric updates. This approach allows the world representation to evolve online with the video stream, reducing feedback latency while preserving geometric fidelity. We instantiate \textbf{Streaming4D} using a Self-Forcing-style autoregressive generator and an incremental reconstruction backend. Experiments show consistent runtime improvements across resolutions on a single RTX 4090 (1.24$\times$ speedup), while maintaining high-quality 4D geometry and multi-view consistency.

Problem

• 4D generation
• interaction latency
• real-time scenarios

Innovation

• block-wise autoregressive video generation
• incremental 3D reconstruction
• synchronous pipeline