GGPT: Geometry Grounded Point Transformer

2026-03-11

AI Summary

This work addresses the geometric inconsistency and detail loss commonly observed in feed-forward 3D reconstruction under sparse-view settings. To this end, it introduces— for the first time—explicit multi-view geometric constraints into a feed-forward framework through a geometry-guided encoding mechanism. The proposed approach integrates an improved dense feature matching-based structure-from-motion (SfM), a lightweight geometric refinement module, and a geometry-supervised point Transformer, jointly enhancing geometric consistency while preserving spatial completeness. Evaluated on ScanNet++, the model significantly outperforms existing feed-forward methods, demonstrating superior capability in reconstructing fine-grained structures and effectively inpainting holes in textureless regions across both in-domain and cross-domain scenarios.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Recent feed-forward networks have achieved remarkable progress in sparse-view 3D reconstruction by predicting dense point maps directly from RGB images. However, they often suffer from geometric inconsistencies and limited fine-grained accuracy due to the absence of explicit multi-view constraints. We introduce the Geometry-Grounded Point Transformer (GGPT), a framework that augments feed-forward reconstruction with reliable sparse geometric guidance. We first propose an improved Structure-from-Motion pipeline based on dense feature matching and lightweight geometric optimisation to efficiently estimate accurate camera poses and partial 3D point clouds from sparse input views. Building on this foundation, we propose a geometry-guided 3D point transformer that refines dense point maps under explicit partial-geometry supervision using an optimised guidance encoding. Extensive experiments demonstrate that our method provides a principled mechanism for integrating geometric priors with dense feed-forward predictions, producing reconstructions that are both geometrically consistent and spatially complete, recovering fine structures and filling gaps in textureless areas. Trained solely on ScanNet++ with VGGT predictions, GGPT generalises across architectures and datasets, substantially outperforming state-of-the-art feed-forward 3D reconstruction models in both in-domain and out-of-domain settings.

Problem

• sparse-view 3D reconstruction
• geometric inconsistency
• fine-grained accuracy
• multi-view constraints
• dense point maps

Innovation

• Geometry-Grounded
• Point Transformer
• Sparse-View 3D Reconstruction
• Geometric Guidance
• Feed-Forward 3D Reconstruction