LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Merging

2025-12-04

AI Summary

To address the prohibitive computational cost and high memory footprint of 3D vision foundation models (e.g., VGGT) when processing long image sequences (hundreds to thousands of frames), this paper proposes a geometry-aware, lightweight decoupling method. The core insight is that local image tokens exhibit strong geometric correlations and cross-layer stability; leveraging this, we design a cache-efficient token merging strategy. We further optimize anchor selection via geometric importance analysis and enable cross-layer reuse of merging indices. The method supports FP8 quantization and seamless fine-tuning of the original model. Experiments demonstrate up to 10× inference speedup and substantial memory compression, while preserving the original model’s accuracy on large-scale scene reconstruction tasks involving up to 1,000 images. The approach is both scalable—adapting to varying sequence lengths—and robust across diverse geometric configurations.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Deep Neural Architectures and Foundation Models
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

3D vision foundation models like Visual Geometry Grounded Transformer (VGGT) have advanced greatly in geometric perception. However, it is time-consuming and memory-intensive for long sequences, limiting application to large-scale scenes beyond hundreds of images. To address this, we propose LiteVGGT, achieving up to 10x speedup and substantial memory reduction, enabling efficient processing of 1000-image scenes. We derive two key insights for 3D reconstruction: (1) tokens from local image regions have inherent geometric correlations, leading to high similarity and computational redundancy; (2) token similarity across adjacent network layers remains stable, allowing for reusable merge decisions. Guided by these, we design a simple yet efficient strategy, dubbed geometry-aware cached token merging. We analyze each token's geometric importance, optimizing anchor token selection to better preserve key information for reconstruction. We also cache and reuse merge indices across layers, substantially reducing latency with minimal accuracy impact. This strategy retains VGGT's core performance, enabling efficient fine-tuning and FP8 quantization for further gains. Extensive experiments validate LiteVGGT's effectiveness, scalability, and robustness. Project page: https://garlicba.github.io/LiteVGGT/

Problem

• Accelerates 3D reconstruction models for large-scale scenes
• Reduces computational redundancy via geometry-aware token merging
• Enables efficient processing of long image sequences

Innovation

• Geometry-aware cached token merging for efficiency
• Optimized anchor token selection preserving key information
• Cached reusable merge indices across network layers