Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding

2025-09-18

AI Summary

This work addresses zero-shot spatio-temporal video grounding (STVG), the task of localizing object trajectories in unseen-category videos solely from textual queries. We propose a decoupled localization framework: first decomposing text queries into attribute and action cues; then introducing a logit-guided re-attention mechanism to explicitly unlock implicit spatio-temporal grounding capabilities embedded in multimodal large language models (MLLMs); and further incorporating spatial–temporal prompt learning and multi-frame temporal ensemble to enhance cross-modal alignment and temporal consistency. The method is fully end-to-end and requires no MLLM fine-tuning. It achieves significant improvements over prior state-of-the-art methods across multiple mainstream STVG benchmarks, including VidSitu, TSG, and GECO. Moreover, it demonstrates strong generalization and architectural agnosticism—delivering consistent performance gains across diverse MLLM backbones (e.g., VideoLLaMA2, Qwen-VL, and InternVL).

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Spatio-temporal video grounding (STVG) aims at localizing the spatio-temporal tube of a video, as specified by the input text query. In this paper, we utilize multimodal large language models (MLLMs) to explore a zero-shot solution in STVG. We reveal two key insights about MLLMs: (1) MLLMs tend to dynamically assign special tokens, referred to as extit{grounding tokens}, for grounding the text query; and (2) MLLMs often suffer from suboptimal grounding due to the inability to fully integrate the cues in the text query ( extit{e.g.}, attributes, actions) for inference. Based on these insights, we propose a MLLM-based zero-shot framework for STVG, which includes novel decomposed spatio-temporal highlighting (DSTH) and temporal-augmented assembling (TAS) strategies to unleash the reasoning ability of MLLMs. The DSTH strategy first decouples the original query into attribute and action sub-queries for inquiring the existence of the target both spatially and temporally. It then uses a novel logit-guided re-attention (LRA) module to learn latent variables as spatial and temporal prompts, by regularizing token predictions for each sub-query. These prompts highlight attribute and action cues, respectively, directing the model's attention to reliable spatial and temporal related visual regions. In addition, as the spatial grounding by the attribute sub-query should be temporally consistent, we introduce the TAS strategy to assemble the predictions using the original video frames and the temporal-augmented frames as inputs to help improve temporal consistency. We evaluate our method on various MLLMs, and show that it outperforms SOTA methods on three common STVG benchmarks. The code will be available at https://github.com/zaiquanyang/LLaVA_Next_STVG.

Problem

• Developing zero-shot spatio-temporal video grounding using MLLMs
• Addressing suboptimal grounding due to incomplete text cue integration
• Improving temporal consistency in spatial attribute predictions

Innovation

• Decomposed spatio-temporal highlighting strategy for query decoupling
• Logit-guided re-attention module for learning spatial-temporal prompts
• Temporal-augmented assembling to improve prediction consistency