LongVU-TTT: Causal Test-Time Training for Visual Resampling in Long Video Understanding

2026-08-26

AI Summary

为了解决长视频理解中视觉令牌预算有限的问题，LongVU-TTT通过在视觉编码器和LLM之间引入因果快速权重更新的卷积测试时训练重采样器来增强模型性能。

Abstract

Long-video MLLMs must model temporal change before a limited visual-token budget removes most frame evidence. We introduce LongVU-TTT, which inserts a convolutional Test-Time Training (TTT) resampler with causal fast-weight updates between the vision encoder and the LLM. Its grouped 2D fast weights adapt to each video and contextualize frame features before compression, while a hybrid uniform-and-change-aware selector retains explicit visual evidence for downstream reasoning. Under controlled conditions, TTT-Conv improves over TTT-MLP by up to +2.12 and bidirectional Mamba2 by up to +3.04 on MLVU, and it is stronger than attention- and fixed-state recurrent resamplers across three benchmarks. Analysis shows that the fast weights behave as a temporal aggregation state rather than a reliable long-horizon episodic memory: their benefit attenuates as evidence becomes more distant, motivating explicit frame retention. LongVU-TTT processes up to 512 frames before reducing them to 128 LLM frames and achieves competitive performance across five video understanding benchmarks.

Problem

• Long Video
• Visual Resampling
• Temporal Change
• Visual-Token Budget
• Frame Evidence

Innovation

• Test-Time Training (TTT)
• causal fast-weight updates
• convolutional resampler
• hybrid selector
• temporal aggregation