Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification

2026-06-27

AI Summary

This work addresses the challenges of high computational cost and frame-level instability in natural language-based spatio-temporal grounding within long videos. The authors propose a second-level object tracking framework that ensures temporal continuity through a cross-second smoothing mechanism and leverages multimodal large language models to generate chain-of-thought trajectories as supervisory signals for reinforcement learning. By integrating denoising training based on ground-truth annotations and a dual-criteria verifier combining temporal Intersection over Union (tIoU) and multi-view IoU (mvIoU), the method significantly reduces computational overhead across varying frame rates while maintaining high localization accuracy, achieving an excellent balance between efficiency and performance.

Abstract

Spatio-temporal grounding in long videos requires precise temporal localization and robust object tracking conditioned on natural-language queries. While recent vision-language models (VLMs) show strong reasoning ability, directly applying frame-by-frame inference to long sequences is computationally expensive and unstable. We propose a practical pipeline that shifts from frame-level to second-level tracking and performs cross-second smoothing to preserve continuity while reducing sequence length. To improve reasoning supervision, we synthesize chain-of-thought style trajectories using advanced multimodal models for temporal localization and target selection, and replace generated spatio-temporal coordinates with ground-truth annotations to avoid noisy supervision. We further optimize the policy with reinforcement learning using a verifier based on $t\_\mathrm{IoU}+mv\_\mathrm{IoU}$. Experiments across multiple FPS settings show that our method achieves a strong trade-off between efficiency and localization quality.

Problem

• spatio-temporal grounding
• long videos
• natural-language queries
• object tracking
• temporal localization

Innovation

• second-level tracking
• multimodal large models
• reinforcement learning verification
• spatio-temporal grounding
• cross-second smoothing