Forget, Anticipate and Adapt: Test Time Training for Long Videos

2026-06-24

AI Summary

Existing test-time training methods incur substantial computational overhead when processing hour-long videos and inefficiently update model parameters in response to negligible inter-frame changes. To address these limitations, this work proposes the Frame Forgetting Network (FFN), which introduces a lightweight test-time training mechanism based on a three-frame sliding window and incorporates a surprise-based metric to dynamically adjust the window size for efficient adaptation. The authors also construct EpicTours, the first dataset comprising three-hour-long videos, and demonstrate FFN’s effectiveness across diverse tasks—including dense segmentation, video classification, and depth estimation—achieving significant reductions in computational cost while successfully extending test-time training to multi-hour video scenarios.

Abstract

Test Time Training (TTT) is a mechanism in which a model adapts to an incoming test-sample by performing some self-supervised (SSL) task and updating its weights even during inference. This procedure does not require labels at test-time. This paper focuses on TTT for long-videos. A major concern with existing approaches is: 1) they perform TTT updates using a sliding window containing frames in the past, whose compute increases linearly with the size of window. This becomes computationally intractable when the videos are hours long. 2) TTT is performed even when temporally close frames look similar, thereby consuming a lot of compute. We present the Frame Forgetting Network (FFN) that: 1) operates on only three frames within the sliding window, namely the frame that exits, the current frame and the frame after that. The model still manages to retain temporal context and work for hours long-videos; 2) mathematically define a surprise metric: how much new information the incoming frame contains with respect to the past seen frame. This facilitates determining how to modify the effective window size during TTT and constitutes the core mechanism of an adaptive windowing algorithm. Additionally, we curate a dataset EpicTours containing up to 3 hour long videos of walking city-tours, whereas earlier datasets on this problem were only 5 min long. We demonstrate FFNs empirical effectiveness on dense-segmentation, video classification tasks, generalization to depth-estimation, and multi-hour long videos.

Problem

• Test Time Training
• long videos
• computational efficiency
• sliding window
• temporal redundancy

Innovation

• Test Time Training
• Long Video
• Adaptive Windowing
• Surprise Metric
• Frame Forgetting Network