How Should Video LLMs Output Time? An Analysis of Efficient Temporal Grounding Paradigms

2026-04-10

AI Summary

This study addresses the tight coupling between output paradigms and model architectures, datasets, and training protocols in existing video temporal localization methods, which hinders isolated evaluation of their impact on performance and lacks systematic analysis of deployment efficiency. Under unified experimental conditions—employing compact vision-language models (SmolVLM2, FastVLM, Molmo2) and LoRA fine-tuning—the work conducts controlled comparisons across three output paradigms: textual numeral generation, temporal token generation, and continuous temporal decoding on Charades-STA, QVHighlights, and YouCook2. Results demonstrate that continuous temporal decoding consistently achieves the best accuracy-efficiency trade-off along the Pareto frontier, maintaining high localization accuracy while significantly reducing inference latency and parameter overhead, thereby offering clear design guidance for edge deployment.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

While Multimodal Large Language Models (MLLMs) have advanced Video Temporal Grounding (VTG), existing methods often couple output paradigms with different backbones, datasets, and training protocols. This makes it challenging to isolate the specific impact of the output design. Additionally, as VTG systems are increasingly considered for resource-constrained edge deployment, the trade-off between output formulation and system-level efficiency requires systematic investigation. In this paper, we present a controlled empirical study comparing three dominant VTG output paradigms: Text Numeral Generation, Temporal Token Generation, and Continuous Temporal Decoding. We evaluate these paradigms across identical compact VLMs (SmolVLM2, FastVLM, and Molmo2) using consistent datasets and LoRA fine-tuning protocols. Evaluations on Charades-STA, QVHighlights, and YouCook2 measure both localization accuracy and system efficiency, including inference latency, training throughput, and parameter overhead. Our results demonstrate that the choice of output formulation significantly affects both grounding accuracy and computational cost, independent of model scale. Specifically, the continuous distribution paradigm consistently achieves the most favorable efficiency-accuracy trade-off on the Pareto frontier, delivering robust localization with minimal latency overhead. These findings provide objective empirical guidelines for designing efficient, deployment-ready VTG systems.

Problem

• Video Temporal Grounding
• Output Paradigm
• Efficiency-Accuracy Trade-off
• Multimodal Large Language Models
• Edge Deployment

Innovation

• Temporal Grounding
• Output Paradigm
• Efficiency-Accuracy Trade-off
• Video LLMs
• Controlled Empirical Study