ReCoSplat: Autoregressive Feed-Forward Gaussian Splatting Using Render-and-Compare

2026-03-10

AI Summary

This work addresses the challenge of robustly reconstructing dynamic scenes from continuous, often pose-free observations in online novel view synthesis. We propose an autoregressive feedforward Gaussian splatting model that incorporates a Render-and-Compare module to provide stable conditioning signals even in the presence of pose errors, effectively mitigating the distributional mismatch between training and inference poses. The framework flexibly supports diverse input settings, including scenarios with or without pose information and camera intrinsics. To handle long observation sequences, we introduce a hybrid key-value cache compression strategy combining early-layer truncation and block-wise selective retention, reducing cache memory by over 90% for sequences exceeding 100 frames. Our method achieves state-of-the-art performance on both in-distribution and out-of-distribution benchmarks.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Online novel view synthesis remains challenging, requiring robust scene reconstruction from sequential, often unposed, observations. We present ReCoSplat, an autoregressive feed-forward Gaussian Splatting model supporting posed or unposed inputs, with or without camera intrinsics. While assembling local Gaussians using camera poses scales better than canonical-space prediction, it creates a dilemma during training: using ground-truth poses ensures stability but causes a distribution mismatch when predicted poses are used at inference. To address this, we introduce a Render-and-Compare (ReCo) module. ReCo renders the current reconstruction from the predicted viewpoint and compares it with the incoming observation, providing a stable conditioning signal that compensates for pose errors. To support long sequences, we propose a hybrid KV cache compression strategy combining early-layer truncation with chunk-level selective retention, reducing the KV cache size by over 90% for 100+ frames. ReCoSplat achieves state-of-the-art performance across different input settings on both in- and out-of-distribution benchmarks. Code and pretrained models will be released. Our project page is at https://freemancheng.com/ReCoSplat .

Problem

• novel view synthesis
• scene reconstruction
• unposed inputs
• Gaussian Splatting
• online reconstruction

Innovation

• Gaussian Splatting
• Render-and-Compare
• autoregressive modeling
• KV cache compression
• novel view synthesis