STAC: Selective Spatiotemporal Aggregation and Compression for Video Reasoning Segmentation

2026-07-02

AI Summary

This work addresses the quadratic computational bottleneck of self-attention in video referring segmentation caused by dense spatiotemporal tokens. It introduces state space models to this task for the first time, proposing a linear-complexity approach for temporal context modeling. By leveraging decoupled bidirectional spatial scanning and causal temporal scanning mechanisms, together with a segmentation-target-driven adaptive thresholding strategy, the method achieves hierarchical and dynamic spatiotemporal token compression. This design preserves semantic completeness while enabling zero-shot streaming inference. Experiments demonstrate that the proposed approach surpasses non-compressed baselines across multiple benchmarks, achieving an 85% reduction in tokens and a 1.8× speedup in inference, effectively balancing efficiency and accuracy.

Abstract

Video reasoning segmentation demands pixel-accurate object tracking across hundreds of frames under complex natural language queries, producing dense spatiotemporal tokens whose quadratic self-attention cost makes long-video processing prohibitive. Existing methods address this through token compression, yet typically operate on encoder features lacking temporal context, constraining selection before content redundancy can be reliably assessed. Informed compression requires contextual awareness, but acquiring that awareness at full resolution incurs the same quadratic cost compression aims to reduce. State-space models resolve this constraint, as their linear recurrence selectively conditions each token on temporal context at $\mathcal{O}(T)$ cost, producing representations where content redundancy becomes assessable. Building on this, Selective SpatioTemporal Aggregation and Compression (STAC) enriches features via decoupled bidirectional spatial and causal temporal scanning, leveraging recurrence-derived redundancy for hierarchical compression with adaptive thresholds optimised with segmentation objective. STAC achieves 85% token reduction and 1.8$\times$ speedup while surpassing compression-free baselines on reasoning segmentation benchmarks in a zero-shot streaming-compatible setting. Code is available \href{https://github.com/MCG-NKU/nku-video}{here}.

Problem

• video reasoning segmentation
• spatiotemporal tokens
• quadratic self-attention cost
• long-video processing
• token compression

Innovation

• state-space models
• spatiotemporal compression
• video reasoning segmentation
• token redundancy
• adaptive hierarchical compression