You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differences

2026-06-14

AI Summary

This work proposes a novel self-supervised visual representation learning paradigm, Temporal Difference in Vision (TDV), which eschews strong inductive biases such as data augmentation, masking, or cropping commonly used in existing methods. Instead, TDV leverages the temporal causal assumption that “the past causes the future” in videos, jointly training an image encoder and a motion encoder so that the sum of the current frame’s representation and the motion representation approximates the representation of the subsequent frame. Relying solely on this weak temporal assumption, the method achieves state-of-the-art performance among self-supervised approaches on dense spatial tasks, demonstrating the effectiveness and potential of modeling temporal causality for large-scale visual representation learning.

Abstract

Progress in AI has largely been driven by methods that assume less. As compute and data increase, approaches with weaker inductive biases generally outperform those with stronger assumptions. This is particularly characteristic of the field of Visual Representation Learning, where approaches have gone from being dominated by Supervised Learning, to Weakly Supervised Learning, to the now widespread success of Self-Supervised Learning without human labels. Yet, even modern Self-Supervised Learning approaches still depend on strong inductive biases such as augmentations, masking, or cropping. If this trend holds, even these remaining biases should become bottlenecks at scale -- and our experiments confirm this: the optimal strength of inductive biases decreases as data grows. This motivates the search for approaches that rely on fewer assumptions. To this end, we introduce Temporal Difference in Vision (TDV), a new paradigm for self-supervised learning from video that avoids existing inductive biases, relying instead on a causal assumption that the past causes the future. TDV functions by jointly training an image encoder and a motion encoder so that the current frame's representation plus the encoded motion equals the next frame's representation. Despite not leveraging any strong inductive biases, TDV matches state-of-the-art recipes on dense spatial tasks, laying the foundation for representation learning without strong assumptions.

Problem

• Visual Representation Learning
• Self-Supervised Learning
• Inductive Biases
• Temporal Differences
• Video

Innovation

• Temporal Difference
• Self-Supervised Learning
• Inductive Bias
• Visual Representation Learning
• Video-based Learning