Unified Spatio-Temporal Token Scoring for Efficient Video VLMs

2026-03-18

AI Summary

This work addresses the significant spatiotemporal redundancy in video vision-language models, where existing token pruning methods struggle to jointly accommodate Vision Transformers (ViTs) and large language models (LLMs) while lacking task adaptability. The authors propose the Spatiotemporal Token Selection (STTS) module, which achieves the first unified, unconditional visual token pruning across both ViT and LLM components. STTS learns inter-frame importance through an auxiliary temporal loss, refines spatial token scores using gradient feedback from the LLM, and incorporates an efficient packing algorithm to enable end-to-end training—all without token merging or text guidance, maintaining a lightweight architecture. Evaluated on 13 video question-answering benchmarks, the method achieves 50% visual token compression and 62% training/inference acceleration with only a 0.7% average performance drop; notably, it even yields a 0.5–1% accuracy gain in long-video scenarios.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Token pruning is essential for enhancing the computational efficiency of vision-language models (VLMs), particularly for video-based tasks where temporal redundancy is prevalent. Prior approaches typically prune tokens either (1) within the vision transformer (ViT) exclusively for unimodal perception tasks such as action recognition and object segmentation, without adapting to downstream vision-language tasks; or (2) only within the LLM while leaving the ViT output intact, often requiring complex text-conditioned token selection mechanisms. In this paper, we introduce Spatio-Temporal Token Scoring (STTS), a simple and lightweight module that prunes vision tokens across both the ViT and the LLM without text conditioning or token merging, and is fully compatible with end-to-end training. By learning how to score temporally via an auxiliary loss and spatially via LLM downstream gradients, aided by our efficient packing algorithm, STTS prunes 50% of vision tokens throughout the entire architecture, resulting in a 62% improvement in efficiency during both training and inference with only a 0.7% drop in average performance across 13 short and long video QA tasks. Efficiency gains increase with more sampled frames per video. Applying test-time scaling for long-video QA further yields performance gains of 0.5-1% compared to the baseline. Overall, STTS represents a novel, simple yet effective technique for unified, architecture-wide vision token pruning.

Problem

• token pruning
• video vision-language models
• computational efficiency
• temporal redundancy
• spatio-temporal redundancy

Innovation

• token pruning
• video VLMs
• spatio-temporal scoring
• efficient inference
• end-to-end training