SToRe3D: Sparse Token Relevance in ViTs for Efficient Multi-View 3D Object Detection

2026-05-13

AI Summary

This work addresses the high inference latency of Vision Transformers in multi-view 3D object detection, which stems from dense processing of 2D image tokens and 3D queries, and the inability of existing sparse methods to jointly optimize both. The authors propose a correlation-aligned sparsification framework that, for the first time, enables joint sparsification of 2D tokens and 3D queries within a ViT architecture. Their approach employs a 2D–3D cross-correlation head to dynamically assess relevance and co-select critical tokens and queries, complemented by a feature caching and reactivation mechanism to reuse filtered features. Evaluated on nuScenes and a newly introduced nuScenes-Relevance benchmark, the method achieves up to 3× speedup with only marginal accuracy degradation, substantially improving inference efficiency and enabling scalable, real-time 3D detection.

Abstract

Vision Transformers (ViTs) enable strong multi-view 3D detection but are limited by high inference latency from dense token and query processing across multiple views and large 3D regions. Existing sparsity methods, designed mainly for 2D vision, prune or merge image tokens but do not extend to full-model sparsity or address 3D object queries. We introduce SToRe3D, a relevance-aligned sparsity framework that jointly selects 2D image tokens and 3D object queries while storing filtered features for reactivation. Mutual 2D-3D relevance heads allocate compute to driving-critical content and preserve other embeddings. Evaluated on nuScenes and our new nuScenes-Relevance benchmark, SToRe3D achieves up to 3x faster inference with marginal accuracy loss, establishing real-time large-scale ViT-based 3D detection while maintaining accuracy on planning-critical agents.

Problem

• 3D object detection
• Vision Transformers
• inference latency
• sparsity
• multi-view

Innovation

• Sparse Token Relevance
• Vision Transformers
• Multi-View 3D Detection
• 3D Object Queries
• Efficient Inference