`Attention-Guided Cross-Temporal Clustering for Self-Supervised Video Object Segmentation

2026-07-08

AI Summary

Unsupervised video object segmentation struggles to simultaneously achieve spatial precision and temporal consistency in complex multi-object scenes. This work proposes a self-supervised, part-level segmentation framework that moves beyond pixel- or whole-object-based modeling by learning part-aware intermediate representations through attention-guided adaptive token selection and lightweight cross-frame clustering. The method leverages a frozen Transformer backbone, multi-offset temporal alignment, and a saliency-weighted symmetric consistency loss—eliminating the need for optical flow, synthetic motion cues, or task-specific pretraining. It achieves efficient, high-accuracy segmentation across diverse resolutions and motion patterns, significantly improving spatiotemporal consistency and generalization capability.

Abstract

Video object segmentation (VOS) is a fundamental task in video understanding, requiring accurate delineation and consistent tracking of objects across frames. While supervised methods achieve strong performance, they rely on densely annotated datasets that are costly to obtain and have limited domain coverage. Self-supervised learning offers a promising alternative by removing the need for manual labels; however, existing approaches often struggle to jointly maintain spatial accuracy and temporal coherence, particularly in unconstrained multi-object scenarios. Many rely on optical flow, synthetic motion cues, or task-specific pretraining, limiting scalability and generalisation. We propose a self-supervised framework, Cross-Temporal Consistency and Clustering, that learns mid-level, part-aware representations by combining attention-guided token selection with lightweight temporal clustering. Instead of operating at the pixel or whole-object level, the method aligns soft part assignments across time using a saliency-weighted symmetric consistency objective. The framework leverages a frozen transformer backbone with lightweight modules for adaptive token selection and multi-offset temporal alignment, enabling efficient scaling across resolutions and motion patterns.

Problem

• video object segmentation
• self-supervised learning
• temporal coherence
• spatial accuracy
• multi-object scenarios

Innovation

• self-supervised learning
• video object segmentation
• attention-guided clustering
• temporal consistency
• part-aware representation