VLM Can Be a Good Assistant: Enhancing Embodied Visual Tracking with Self-Improving Visual-Language Models

2025-05-27

AI Summary

To address the challenge of autonomous recovery after tracking failure in embodied visual tracking, this paper proposes the first self-improving framework integrating off-the-shelf active trackers with vision-language models (VLMs). Under normal conditions, it employs a low-latency visual tracking strategy for efficiency; upon failure detection, it triggers VLM-driven reasoning augmented by a memory-enhanced self-reflection mechanism to mitigate VLMs’ inherent limitations in 3D spatial reasoning, enabling dynamic synergy between VLM and visual strategies. Key contributions include: (i) the first integration of VLMs into active failure recovery for embodied visual tracking; (ii) a novel memory-augmented self-reflection mechanism; and (iii) a robust failure-detection and seamless switching module. Evaluated in challenging, dynamic, unstructured environments, our method achieves a 72% higher success rate than state-of-the-art reinforcement learning approaches and a 220% improvement over PID-based baselines, significantly enhancing robustness in long-term target monitoring.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

We introduce a novel self-improving framework that enhances Embodied Visual Tracking (EVT) with Visual-Language Models (VLMs) to address the limitations of current active visual tracking systems in recovering from tracking failure. Our approach combines the off-the-shelf active tracking methods with VLMs' reasoning capabilities, deploying a fast visual policy for normal tracking and activating VLM reasoning only upon failure detection. The framework features a memory-augmented self-reflection mechanism that enables the VLM to progressively improve by learning from past experiences, effectively addressing VLMs' limitations in 3D spatial reasoning. Experimental results demonstrate significant performance improvements, with our framework boosting success rates by $72%$ with state-of-the-art RL-based approaches and $220%$ with PID-based methods in challenging environments. This work represents the first integration of VLM-based reasoning to assist EVT agents in proactive failure recovery, offering substantial advances for real-world robotic applications that require continuous target monitoring in dynamic, unstructured environments. Project website: https://sites.google.com/view/evt-recovery-assistant.

Problem

• Enhancing Embodied Visual Tracking with self-improving VLMs
• Addressing tracking failure recovery in active visual systems
• Improving 3D spatial reasoning in Visual-Language Models

Innovation

• Combines active tracking with VLM reasoning
• Uses memory-augmented self-reflection for improvement
• Activates VLM only upon failure detection