XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression

2026-01-03

arXiv.org

AI Summary

This work addresses the unbounded growth of key-value (KV) cache in StreamVGGT for streaming 3D reconstruction, which leads to excessive memory consumption and latency. To mitigate this issue, the authors propose a fine-tuning-free joint pruning and quantization compression framework. By analyzing multi-view redundancy, the method efficiently estimates token importance and designs a quantization strategy tailored to the distribution characteristics of KV tensors, enabling precise pruning and compression under the causal attention mechanism. Experimental results demonstrate that the proposed approach reduces memory usage by 4.42× and accelerates inference by 5.48×, with negligible degradation in reconstruction quality, thereby significantly enhancing the scalability and practicality of streaming 3D reconstruction systems.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Learning on the Edge & Model Compression
• Data Mining & Knowledge Management: Data Compression

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Learning-based 3D visual geometry models have benefited substantially from large-scale transformers. Among these, StreamVGGT leverages frame-wise causal attention for strong streaming reconstruction, but suffers from unbounded KV cache growth, leading to escalating memory consumption and inference latency as input frames accumulate. We propose XStreamVGGT, a tuning-free approach that systematically compresses the KV cache through joint pruning and quantization, enabling extremely memory-efficient streaming inference. Specifically, redundant KVs originating from multi-view inputs are pruned through efficient token importance identification, enabling a fixed memory budget. Leveraging the unique distribution of KV tensors, we incorporate KV quantization to further reduce memory consumption. Extensive evaluations show that XStreamVGGT achieves mostly negligible performance degradation while substantially reducing memory usage by 4.42$\times$ and accelerating inference by 5.48$\times$, enabling scalable and practical streaming 3D applications. The code is available at https://github.com/ywh187/XStreamVGGT/.

Problem

• KV cache growth
• memory efficiency
• streaming 3D reconstruction
• inference latency
• vision geometry

Innovation

• KV cache compression
• streaming 3D reconstruction
• memory-efficient transformer
• token pruning
• KV quantization