Your VLM Already Knows When: Training-Free Temporal Grounding by Asking Yes or No

2026-08-08

AI Summary

Existing vision-language models often fail at temporal event localization due to suboptimal task interface design, frequently producing highly confident yet inaccurate time predictions despite correctly recognizing the underlying events. To address this, this work proposes a training-free temporal localization framework that reformulates timestamp regression as a coarse-to-fine binary question-answering scan over the video timeline and ranks candidate intervals using first-token probabilities. Relying solely on inference-time design without any temporal supervision, the method achieves 56.8% R@0.5 on Charades-STA, outperforming all existing training-free approaches and even several supervised methods. It also surpasses all trained TVG models under zero-shot evaluation on TACoS and demonstrates substantial improvements on ActivityNet Captions and QVHighlights.

Abstract

Multimodal LLMs that recognise events reliably still fail to say when they happen. Prompted for timestamps, strong VLMs reach as little as $3.8\%$ R@0.5 on Charades-STA, and $77$ to $80\%$ of their wrong predictions carry low output entropy: the models are confidently wrong, and entropy-based error detection stays below a random classifier. We show that this failure lives in the task interface, not in perception. Holding the weights fixed, replacing timestamp regression with a coarse-to-fine scan of binary questions, whose first-token probabilities are consumed only as a ranking, raises R@0.5 by $28$ to $50$ points across four frozen backbones. The residual failures decompose into two measurable axes: a perception axis that moves with the backbone, and a geometry axis that is analytically predictable from the ratio of the output-window and event widths. FV-Action, the training-free method built on this analysis, reaches $56.8\%$ R@0.5 on Charades-STA, above the same backbone's native grounding pipeline and the strongest training-free result on this benchmark; it surpasses every TVG-trained model evaluated zero-shot on TACoS, and improves over direct prediction on ActivityNet Captions and QVHighlights, with no temporal supervision at any stage.

Problem

• temporal grounding
• visual language models
• timestamp prediction
• multimodal LLMs
• time localization

Innovation

• training-free
• temporal grounding
• binary questioning
• frozen VLM
• zero-shot