DreamStereo: Towards Real-Time Stereo Inpainting for HD Videos

2026-04-14

AI Summary

This work addresses the challenge of achieving high-quality, real-time inpainting in stereo videos, where disparity-induced occlusions result in sparse and discontinuous regions, and existing approaches are hindered by limited dataset quality and excessive global redundant computation. To overcome these limitations, we propose a geometry-guided, sparsity-aware inpainting framework that integrates Gradient-Aware Disparity Warping (GAPW), Parallax-Based Dual Projection (PBDP), and Sparsity-Aware Stereo Inpainting (SASI). Our method generates temporally coherent and geometrically accurate results without requiring ground-truth stereo video data. Implemented on a single A100 GPU, it achieves real-time performance at 25 FPS for 768×1280 resolution, offering a 10.7× speedup in inference and reducing computational redundancy by over 70%, while maintaining inpainting quality comparable to full-computation baselines.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Stereo video inpainting, which aims to fill the occluded regions of warped videos with visually coherent content while maintaining temporal consistency, remains a challenging open problem. The regions to be filled are scattered along object boundaries and occupy only a small fraction of each frame, leading to two key challenges. First, existing approaches perform poorly on such tasks due to the scarcity of high-quality stereo inpainting datasets, which limits their ability to learn effective inpainting priors. Second, these methods apply equal processing to all regions of the frame, even though most pixels require no modification, resulting in substantial redundant computation. To address these issues, we introduce three interconnected components. We first propose Gradient-Aware Parallax Warping (GAPW), which leverages backward warping and the gradient of the coordinate mapping function to obtain continuous edges and smooth occlusion regions. Then, a Parallax-Based Dual Projection (PBDP) strategy is introduced, which incorporates GAPW to produce geometrically consistent stereo inpainting pairs and accurate occlusion masks without requiring stereo videos. Finally, we present Sparsity-Aware Stereo Inpainting (SASI), which reduces over 70% of redundant tokens, achieving a 10.7x speedup during diffusion inference and delivering results comparable to its full-computation counterpart, enabling real-time processing of HD (768 x 1280) videos at 25 FPS on a single A100 GPU.

Problem

• stereo video inpainting
• occlusion filling
• temporal consistency
• parallax warping
• HD video processing

Innovation

• stereo video inpainting
• gradient-aware parallax warping
• parallax-based dual projection
• sparsity-aware inference
• real-time HD video processing