Temporally Consistent Object-Centric Learning by Contrasting Slots

2024-12-18

arXiv.org

AI Summary

This work addresses the lack of long-term temporal consistency in unsupervised video object-centric learning by introducing a novel object-level temporal contrastive loss that explicitly enforces stability of object slots across time. Built upon the Slot Attention architecture, the method integrates object-level temporal contrastive learning, cross-frame slot matching, and consistency regularization—entirely without supervision—to enhance the temporal robustness of object representations in dynamic scenes. Evaluated on both synthetic (CATER, PHYRE) and real-world video datasets (KTH, BAIR), the approach achieves state-of-the-art performance: it yields more reliable video object decomposition and significantly reduces unsupervised object dynamics prediction error. By producing stable, structured, and temporally coherent object representations, the method establishes a stronger foundation for downstream tasks such as autonomous control.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Unsupervised & Self-Supervised Learning
• Planning, Routing, and Scheduling: Temporal Planning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Unsupervised object-centric learning from videos is a promising approach to extract structured representations from large, unlabeled collections of videos. To support downstream tasks like autonomous control, these representations must be both compositional and temporally consistent. Existing approaches based on recurrent processing often lack long-term stability across frames because their training objective does not enforce temporal consistency. In this work, we introduce a novel object-level temporal contrastive loss for video object-centric models that explicitly promotes temporal consistency. Our method significantly improves the temporal consistency of the learned object-centric representations, yielding more reliable video decompositions that facilitate challenging downstream tasks such as unsupervised object dynamics prediction. Furthermore, the inductive bias added by our loss strongly improves object discovery, leading to state-of-the-art results on both synthetic and real-world datasets, outperforming even weakly-supervised methods that leverage motion masks as additional cues.

Problem

• Enhance temporal consistency in object-centric video learning.
• Improve object discovery and representation stability across frames.
• Facilitate unsupervised object dynamics prediction in videos.

Innovation

• Object-level temporal contrastive loss introduced
• Enhances temporal consistency in video representations
• Improves object discovery and dynamics prediction