RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer

2026-06-16

AI Summary

This work addresses the limited scalability of existing Visual Geometry Grounded Transformers (VGGTs) due to their high computational complexity in cross-frame attention and the oversight of functional heterogeneity across network layers in current acceleration approaches. The study is the first to reveal that shallow, intermediate, and deep layers of VGGTs respectively perform structure perception, geometric aggregation, and pose refinement in multi-view geometry reconstruction. Building on this insight, the authors propose a dual-axis non-uniform compression strategy that integrates training-free saliency-guided band merging with selectively preserved key/value downsampling. This is further enhanced by phase-shifted spatial grids, reference-frame anchors, and uncompressed camera/register tokens, which collectively preserve critical spatial structures while drastically reducing redundant computation. Experiments demonstrate a 6.7× inference speedup with negligible degradation in reconstruction quality.

Abstract

Visual Geometry Grounded Transformer (VGGT) recovers dense 3D scene structure from multi-view images in one forward pass, but quadratic cross-frame attention limits its scalability. Existing training-free accelerators reduce computation uniformly along one axis, missing layer heterogeneity. Our spectral, probing, and causal analyses reveal three regimes: shallow layers lack cross-view structure, middle layers drive cross-view alignment, and deep layers are redundant for dense geometry yet their cross-frame attention remains essential for pose. RegimeVGGT applies layer-wise U-shaped compression along two axes: Saliency-Guided Banded Merging protects geometry- and edge-salient tokens, while Selectively Protected K/V Downsampling preserves cross-frame spatial coverage and the pose-critical path through a phase-shifted spatial grid, a reference-frame anchor, and uncompressed camera/register tokens. Training-free, RegimeVGGT achieves a 6.7x speedup over VGGT* at matched reconstruction quality.

Problem

• Visual Geometry Grounded Transformer
• cross-frame attention
• scalability
• layer heterogeneity
• dense 3D reconstruction

Innovation

• layer-wise compression
• spatially preserving redundancy removal
• training-free acceleration
• cross-frame attention
• Visual Geometry Grounded Transformer