Leveraging Text-to-Image Diffusion Models for Unsupervised Visual Object Tracking

2026-05-26

AI Summary

This work addresses the limited exploitation of semantic and structural information in unsupervised visual object tracking by introducing, for the first time, a pretrained text-to-image diffusion model into this task. The proposed approach establishes a novel tracking paradigm based on prompt learning and online updating, wherein cross-attention mechanisms enable the model to adaptively learn an initial target-specific prompt without any annotations and dynamically refine it during inference. This facilitates accurate target localization and robust tracking performance. Extensive experiments on six challenging benchmark datasets demonstrate the effectiveness of the method, which significantly outperforms existing unsupervised trackers and highlights the potential of diffusion models in cross-modal semantic understanding and unsupervised tracking scenarios.

Abstract

Unsupervised visual object tracking is a challenging task that requires following arbitrary targets in videos without training on ground-truth annotations. Despite considerable progress, existing state-of-the-art unsupervised trackers often struggle in scenarios that demand fine-grained understanding of semantic and visual structural information within video frames. Text-to-image diffusion models are well known for their ability to generate images that accurately reflect the semantics and structures described in the input prompt, demonstrating a strong grasp of visual semantics and structures. Building on this capability, we approach the unsupervised tracking from a new perspective by exploiting the rich semantic knowledge encoded in pretrained text-to-image diffusion models. To adapt the diffusion models, which are originally developed for image generation, to the tracking task, we reinterpret the models as a bridge between text and image modalities. This connection is realized through the cross-attention mechanism: when both text and an image are input into the models, they highlight the regions of the image that are semantically aligned with the text in the cross-attention maps. We therefore learn a prompt that represents the tracking target and activates its corresponding region in the cross-attention map for each frame, which enables object tracking with the diffusion model. Specifically, our method Diff-Tracking is composed of two main components: an initial prompt learner and an online prompt updater. The initial prompt learner generates a prompt that captures the target object in the first frame, allowing the diffusion model to identify the target. The online prompt updater refines the prompt based on motion information, enabling consistent tracking across video frames. We evaluate our approach on six challenging tracking datasets demonstrate the effectiveness of our approach.

Problem

• Unsupervised visual object tracking
• Text-to-image diffusion models
• Semantic understanding
• Visual structure
• Cross-attention mechanism

Innovation

• diffusion models
• unsupervised tracking
• cross-attention
• prompt learning
• text-to-image generation