Burn After Reading: Do Multimodal Large Language Models Truly Capture Order of Events in Image Sequences?

2025-06-12

AI Summary

Can multimodal large language models (MLLMs) accurately comprehend temporal relationships among events in image sequences? This paper introduces TempVS, the first benchmark dedicated to fine-grained temporal reasoning, comprising three tasks—event relation inference, sentence ordering, and image ordering—that jointly demand spatiotemporal localization and cross-modal reasoning over visual and linguistic inputs. TempVS innovatively incorporates foundational localization tests, establishing a hierarchical evaluation framework: “localization → relation → ordering.” A systematic assessment across 38 state-of-the-art MLLMs reveals pervasive and substantial deficits in temporal understanding, with performance markedly lagging behind human capabilities. We provide granular failure analysis and publicly release the full dataset, implementation code, and evaluation framework to establish a foundational benchmark for advancing research on temporal cognition in MLLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

This paper introduces the TempVS benchmark, which focuses on temporal grounding and reasoning capabilities of Multimodal Large Language Models (MLLMs) in image sequences. TempVS consists of three main tests (i.e., event relation inference, sentence ordering and image ordering), each accompanied with a basic grounding test. TempVS requires MLLMs to rely on both visual and linguistic modalities to understand the temporal order of events. We evaluate 38 state-of-the-art MLLMs, demonstrating that models struggle to solve TempVS, with a substantial performance gap compared to human capabilities. We also provide fine-grained insights that suggest promising directions for future research. Our TempVS benchmark data and code are available at https://github.com/yjsong22/TempVS.

Problem

• Assessing MLLMs' ability to understand event order in image sequences
• Evaluating temporal grounding and reasoning in multimodal language models
• Identifying performance gaps between MLLMs and humans in temporal tasks

Innovation

• Introduces TempVS benchmark for MLLMs
• Tests event relation and ordering capabilities
• Evaluates 38 models revealing performance gaps