USS: Unified Spatial-Semantic Prompts for Embodied Visual Tracking with Latent Dynamics Learning

2026-06-24

AI Summary

This work addresses the challenge of instance-level identity ambiguity in embodied visual tracking, where existing methods relying solely on textual prompts struggle to distinguish targets from semantically similar distractors. To overcome this limitation, the authors propose USS, a unified spatial-semantic prompting paradigm that, for the first time, integrates spatial cues—such as points, bounding boxes, and masks—with textual prompts within an end-to-end multimodal tracking framework. USS enhances temporal consistency and robustness through modality-specific encoders, a hybrid attention mechanism, an egocentric waypoint decoder, and a self-supervised latent dynamic world model. Experiments demonstrate that USS significantly outperforms text-only approaches in real-world robotic scenarios, achieves state-of-the-art performance among non-MLLM methods on simulation benchmarks, and matches the accuracy of the latest MLLM-based trackers with substantially higher inference efficiency.

Abstract

Embodied Visual Tracking (EVT) requires an agent to continuously follow a specified target while actively moving through dynamic environments. However, prevailing EVT paradigms predominantly rely on language-based target indication. While language is expressive and convenient, cluttered scenes often contain multiple objects that satisfy the same semantic description, leading to ambiguous target grounding. We therefore propose a paradigm shift, reframing target indication in EVT from text-only specification to unified spatial-semantic prompting. Based on this paradigm, we introduce Unified Spatial-Semantic Prompts for Embodied Visual Tracking with Latent Dynamics Learning, USS, an end-to-end embodied tracking framework that supports text, point, bounding box, and mask prompts within a unified architecture. USS encodes heterogeneous prompts with modality-specific encoders, fuses prompt tokens with visual features through hybrid attention, and decodes compact prompt-conditioned representations into egocentric waypoints. To further improve temporal robustness, USS incorporates a latent world model that predicts future representations through self-supervised alignment. Real-robot experiments demonstrate that explicit spatial target cues yield higher success rates than text-only prompts, particularly in scenarios involving similar distractors and longer-horizon tracking where maintaining instance-level target identity is critical. In the simulation benchmark, USS also achieves state-of-the-art performance among non-MLLM-based methods and competitive results against recent MLLM-based approaches with faster inference speed. Our findings reveal that spatial-semantic prompting provides a more precise and flexible target indication interface for embodied visual tracking. Project site: https://arescheah.github.io/uss-project-page/.

Problem

• Embodied Visual Tracking
• target grounding
• spatial-semantic prompting
• ambiguous reference
• instance-level identity

Innovation

• Unified Spatial-Semantic Prompts
• Embodied Visual Tracking
• Latent Dynamics Learning
• Multimodal Prompting
• Egocentric Waypoint Prediction