Efficient Dense Matching for Enhanced Gaussian Splatting Using AV1 Motion Vectors

2026-05-14

AI Summary

This work addresses the limitations of existing 3D Gaussian Splatting (3DGS) methods, which rely on sparse initial point clouds and suffer from the high computational cost and poor robustness of traditional Structure-from-Motion (SfM) pipelines in textureless regions. The authors propose a novel initialization strategy that leverages motion vectors embedded in AV1 video encoders to replace time-consuming feature matching, enabling a lightweight and efficient pipeline for dense point cloud generation. This approach significantly increases input point cloud density—up to eight times that of conventional methods—while preserving geometric accuracy. Consequently, reconstruction quality, measured by VMAF, improves by 9 points, and the average training time required to reach baseline quality is reduced by 63%.

Abstract

3D Gaussian Splatting (3DGS) has emerged as a prominent framework for real-time, photorealistic scene reconstruction, offering significant speed-ups over Neural Radiance Fields (NeRF). However, the fidelity of 3DGS representations remains heavily dependent on the quality of the initial point cloud. While standard Structure-from-Motion (SfM) pipelines using COLMAP provide adequate initialisation, they often suffer from high computational costs and sparsity in textureless regions, which degrades subsequent reconstruction accuracy and convergence speed. In this work, we introduce an AV1-based feature detection and matching pipeline that significantly reduces SfM processing overhead. By leveraging motion vectors inherent to the AV1 video codec, we bypass computationally expensive exhaustive matching while maintaining geometric robustness. Our pipeline produces substantially denser point clouds, with up to eight times as many points as classical SfM. We demonstrate that this enhanced initialisation directly improves 3DGS performance, yielding an 9-point increase in VMAF and a 63% average reduction in training time required to reach baseline quality. The project page: https://sigmedia.tv/AV1-3DGS.github.io/

Problem

• 3D Gaussian Splatting
• Structure-from-Motion
• point cloud initialization
• textureless regions
• computational cost

Innovation

• AV1 motion vectors
• dense point cloud
• 3D Gaussian Splatting
• efficient feature matching
• Structure-from-Motion