Learning Video Dynamics with Predictive Differentiable Rendering

2026-06-29

AI Summary

Existing video prediction models operating in discrete pixel space often rely on mean squared error (MSE) loss, which tends to produce overly smoothed predictions with diminished detail. To address this limitation, this work proposes a novel Predictive Differentiable Rendering (PDR) paradigm that introduces 2D Gaussian differentiable rendering into video prediction for the first time. The approach balances continuous representation and discrete prediction through a lightweight, plug-and-play adapter, PredGS, coupled with an efficient CUDA-accelerated renderer, predgsplat. By jointly optimizing L1 loss and structural similarity (SSIM), the method significantly enhances visual fidelity, detail preservation, and prediction accuracy across multiple benchmarks—including TaxiBJ, WeatherBench, KTH, and Human3.6M—while achieving up to a 10× speedup in rendering compared to baseline methods.

Abstract

How to accurately predict a high-fidelity future world? While the visual world is inherently continuous, existing deterministic video prediction models operate in discrete pixel space and are mainly optimized with pixel-wise mean squared error (MSE), which often leads to over-smoothed predictions and a lack of fine-grained visual details. To address these limitations, we propose Predictive Differentiable Rendering (PDR), a novel end-to-end video prediction paradigm that bridges the gap between discrete and continuous representations. Inspired by recent progress in 3D reconstruction with 3D Gaussian Splatting, we introduce PredGS, a lightweight and plug-and-play adapter based on 2D Gaussian representation, which could be seamlessly integrated with existing pixel space predictors, significantly improving spatial detail preservation with negligible computational overhead. Furthermore, we develop predgsplat, a CUDA-accelerated differentiable 2D Gaussian renderer supporting arbitrary channels. Each Gaussian is defined by 5 + C learnable parameters (position, scale, rotation, and C channel amplitudes) and achieves up to 10x faster rendering than the baseline. Optimized by a combined L1 and SSIM loss, PDR overcomes the inherent blurring tendencies of MSE Loss, significantly enhancing the prediction performance. Extensive experiments on diverse real-world benchmarks, including TaxiBJ, WeatherBench, KTH, and Human3.6M, demonstrate that PDR consistently surpasses existing methods, delivering superior detail preservation, visual fidelity, and predictive accuracy.

Problem

• video prediction
• over-smoothed predictions
• visual detail
• pixel-wise MSE
• high-fidelity future

Innovation

• Predictive Differentiable Rendering
• 2D Gaussian Splatting
• Video Prediction
• Differentiable Rendering
• Detail Preservation