TIME: Temporal-sensitive Multi-dimensional Instruction Tuning and Benchmarking for Video-LLMs

2025-03-13

AI Summary

Video-LLMs exhibit significant limitations in temporal reasoning tasks. To address this, we propose a systematic solution comprising three core components: (1) the first five-dimensional temporal-aware instruction-tuning dataset—covering sequence order, duration, causality, frequency, and relative timing; (2) a multi-task prompt-tuning framework that requires no additional temporal annotations; and (3) a novel temporal understanding benchmark designed to be robust against spatial/static shortcut biases and enforce multi-dimensional alignment. Our method integrates temporal dimension disentanglement modeling with shortcut identification and filtering. Extensive experiments demonstrate that our approach achieves an average 19.7% improvement across temporal reasoning, event ordering, and duration estimation tasks, substantially enhancing the robustness and fidelity of Video-LLMs’ temporal modeling capabilities.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Video large language models have achieved remarkable performance in tasks such as video question answering, however, their temporal understanding remains suboptimal. To address this limitation, we curate a dedicated instruction fine-tuning dataset that focuses on enhancing temporal comprehension across five key dimensions. In order to reduce reliance on costly temporal annotations, we introduce a multi-task prompt fine-tuning approach that seamlessly integrates temporal-sensitive tasks into existing instruction datasets without requiring additional annotations. Furthermore, we develop a novel benchmark for temporal-sensitive video understanding that not only fills the gaps in dimension coverage left by existing benchmarks but also rigorously filters out potential shortcuts, ensuring a more accurate evaluation. Extensive experimental results demonstrate that our approach significantly enhances the temporal understanding of video-LLMs while avoiding reliance on shortcuts.

Problem

• Enhance temporal understanding in video large language models.
• Reduce reliance on costly temporal annotations in video tasks.
• Develop a benchmark for accurate temporal-sensitive video understanding.

Innovation

• Temporal-sensitive instruction fine-tuning dataset
• Multi-task prompt fine-tuning without annotations
• Novel benchmark for temporal video understanding