3DTV: A Feedforward Interpolation Network for Real-Time View Synthesis

2026-04-13

AI Summary

This work addresses the challenge of real-time free-viewpoint rendering by balancing multi-camera data redundancy with low-latency interactivity. To this end, the authors propose 3DTV, a feed-forward sparse-view interpolation network that requires no scene-specific optimization. The method integrates lightweight geometric priors with learned strategies, employing Delaunay triplets for view selection and a pose-aware depth estimation module to circumvent explicit proxy representations. Robust cross-scene rendering is achieved through a coarse-to-fine depth pyramid, feature reprojection, and occlusion-aware fusion. Experiments demonstrate that 3DTV significantly outperforms existing real-time novel view synthesis approaches on multi-view video datasets, achieving an excellent trade-off between rendering quality and computational efficiency, making it well-suited for latency-sensitive applications such as AR/VR.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Real-time free-viewpoint rendering requires balancing multi-camera redundancy with the latency constraints of interactive applications. We address this challenge by combining lightweight geometry with learning and propose 3DTV, a feedforward network for real-time sparse-view interpolation. A Delaunay-based triplet selection ensures angular coverage for each target view. Building on this, we introduce a pose-aware depth module that estimates a coarse-to-fine depth pyramid, enabling efficient feature reprojection and occlusion-aware blending. Unlike methods that require scene-specific optimization, 3DTV runs feedforward without retraining, making it practical for AR/VR, telepresence, and interactive applications. Our experiments on challenging multi-view video datasets demonstrate that 3DTV consistently achieves a strong balance of quality and efficiency, outperforming recent real-time novel-view baselines. Crucially, 3DTV avoids explicit proxies, enabling robust rendering across diverse scenes. This makes it a practical solution for low-latency multi-view streaming and interactive rendering. Project Page: https://stefanmschulz.github.io/3DTV_webpage/

Problem

• real-time view synthesis
• free-viewpoint rendering
• sparse-view interpolation
• low-latency rendering
• multi-view video

Innovation

• real-time view synthesis
• feedforward interpolation
• pose-aware depth estimation
• sparse-view rendering
• occlusion-aware blending