Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking

2026-06-28

AI Summary

Existing vision-language tracking methods are prone to erroneous updates, background interference, and model hallucination when the target’s appearance and location undergo dynamic changes, primarily due to semantic-visual misalignment. This work proposes the first approach that integrates linguistic dependency parsing into vision-language tracking, enabling structured extraction of key components—such as the target object, semantic concepts, and background context—from natural language instructions. By leveraging the cross-modal comprehension capabilities of Qwen-VL, the method achieves component-aware adaptive updating of textual descriptions. This strategy effectively suppresses hallucination and distractor interference, establishing state-of-the-art performance across multiple benchmarks, including TNL2K, LaSOT, TNLLT, and OTB-LANG.

Abstract

Vision-language tracking guided by natural language specifications leverages high-level semantic cues of target objects to substantially boost tracking accuracy and robustness. Existing studies have verified that adaptively optimizing textual descriptions throughout the tracking process can effectively mitigate the semantic-visual mismatch induced by dynamic variations in target appearance, position, and other inherent attributes. Nevertheless, mainstream methods that directly generate textual information via sequence models or large language models inevitably suffer from inherent defects, including erroneous target updating, excessive background distraction, and pervasive hallucination artifacts. To address the aforementioned limitations, this paper proposes a novel language dependency parsing mechanism to precisely distill core tracking principal components, encompassing target objects, semantic concepts, and background contextual information. On this basis, we perform component-aware adaptive textual description updates by exploiting the powerful cross-modal understanding capability of the pre-trained vision-language model Qwen-VL. By integrating the proposed elaborately designed modules into the baseline framework, our method achieves consistent and superior tracking performance on multiple large-scale vision-language tracking benchmarks, including TNL2K, LaSOT, TNLLT, and OTB-LANG. The source code and pre-trained models will be released at https://github.com/Event-AHU/Open_VLTrack.

Problem

• vision-language tracking
• natural language specification
• semantic-visual mismatch
• textual description update
• hallucination

Innovation

• language dependency parsing
• vision-language tracking
• adaptive textual update
• cross-modal understanding
• Qwen-VL