Diversity-aware View Partitioning for Scalable VGGT

2026-07-02

AI Summary

This work addresses the limitations of Vision-Guided Geometry Transformers (VGGTs) when scaling to large-scale multi-view scenes, where quadratic computational complexity of attention mechanisms and sensitivity to view distribution cause redundant views to dilute geometric information. To overcome these issues, the authors propose a training-free, plug-and-play inference framework that introduces a diversity-aware view chunking strategy. This strategy organizes input views into balanced, informative chunks by jointly considering visual dissimilarity and spatial dispersion through combinatorial graph partitioning. Additionally, soft pose propagation is employed to estimate spatial relationships, guiding the Transformer to attend to salient geometric cues while reducing redundant attention interactions. The method significantly improves performance in camera pose estimation, depth prediction, and 3D reconstruction, while simultaneously lowering memory consumption and inference latency, and remains compatible with existing VGGT variants.

Abstract

Geometry transformers such as VGGT achieve strong performance by jointly reasoning over multiple views with global attention. However, scaling them to large view collections remains challenging due to the quadratic cost of attention. Moreover, our empirical analysis reveals that the reconstruction quality in VGGT is sensitive to the distribution of viewpoints. Simply increasing the number of views without sufficient viewpoint diversity can even degrade performance, as redundant views introduce highly similar tokens that dilute informative geometric signals in the attention mechanism. Motivated by this observation, we propose a training-free and plug-and-play VGGT inference framework that organizes views into diversity-aware balanced chunks. The chunks are constructed through combinatorial graph partitioning over visual dissimilarity and spatial dispersion. This view organization allows the transformer to focus attention on geometrically informative views while reducing redundant attention interactions. To estimate spatial dispersion without full pose estimation, we approximate spatial relationships via a soft pose propagation strategy based on visual similarity from a small set of seed frames. Extensive experiments demonstrate improved performance in camera pose estimation, multi-view depth prediction, and 3D reconstruction while reducing memory usage and inference latency. Our framework also complements existing VGGT variants, enabling scalable multi-view reconstruction without sacrificing geometric fidelity.

Problem

• view diversity
• scalable transformers
• multi-view reconstruction
• attention redundancy
• geometric fidelity

Innovation

• diversity-aware partitioning
• scalable transformers
• view redundancy reduction
• soft pose propagation
• multi-view reconstruction