Smoothing Slot Attention Iterations and Recurrences

2025-08-07

AI Summary

Slot Attention (SA) faces two key challenges in object-centric learning: (1) cold-start queries for the first frame of images/videos lack sample-specificity, leading to imprecise object aggregation; and (2) subsequent video frames reuse the same query update strategy as the first frame, ignoring their inherent sample-awareness. This paper proposes SmoothSA, the first SA variant to introduce a self-distillation-based query warm-up mechanism that leverages image content to enhance initial queries for the first frame. Additionally, we design a heterogeneous iterative strategy: three full iterations for the first frame versus a single refinement step for subsequent frames—tailored to their distinct representational properties. SmoothSA significantly improves object separation accuracy and temporal consistency, outperforming existing SA variants across multiple object discovery, recognition, and downstream tasks.

Technology Category

• Search and Optimization: Learning to Search
• Computer Vision: Motion & Tracking
• Machine Learning: Semi-Supervised Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Slot Attention (SA) and its variants lie at the heart of mainstream Object-Centric Learning (OCL). Objects in an image can be aggregated into respective slot vectors, by extit{iteratively} refining cold-start query vectors, typically three times, via SA on image features. For video, such aggregation is extit{recurrently} shared across frames, with queries cold-started on the first frame while transitioned from the previous frame's slots on non-first frames. However, the cold-start queries lack sample-specific cues thus hinder precise aggregation on the image or video's first frame; Also, non-first frames' queries are already sample-specific thus require transforms different from the first frame's aggregation. We address these issues for the first time with our extit{SmoothSA}: (1) To smooth SA iterations on the image or video's first frame, we extit{preheat} the cold-start queries with rich information of input features, via a tiny module self-distilled inside OCL; (2) To smooth SA recurrences across all video frames, we extit{differentiate} the homogeneous transforms on the first and non-first frames, by using full and single iterations respectively. Comprehensive experiments on object discovery, recognition and downstream benchmarks validate our method's effectiveness. Further analyses intuitively illuminate how our method smooths SA iterations and recurrences. Our code is available in the supplement.

Problem

• Cold-start queries lack sample-specific cues for precise aggregation.
• Non-first frames require different transforms from first frame's aggregation.
• SmoothSA preheats queries and differentiates transforms for better performance.

Innovation

• Preheat cold-start queries with input features
• Differentiate transforms for first and non-first frames
• Self-distilled tiny module for query preheating