VIR-Bench: Evaluating Geospatial and Temporal Understanding of MLLMs via Travel Video Itinerary Reconstruction

2025-09-23

AI Summary

Existing video understanding benchmarks predominantly focus on indoor or short-range scenarios, failing to assess multimodal large language models’ (MLLMs) capability to comprehend complex spatiotemporal trajectories encountered during long-distance travel. Method: To address this gap, we introduce VIR-Bench—the first benchmark dedicated to long-distance travel—comprising 200 authentic travel videos and a novel itinerary reconstruction task designed to systematically evaluate MLLMs’ structured reasoning across diverse spatial and temporal scales. We propose a fine-grained spatiotemporal alignment evaluation framework and, informed by empirical insights, develop a travel planning prototype agent. Results: Experiments reveal limited performance of state-of-the-art MLLMs on this task, confirming its challenge; our enhanced agent significantly improves itinerary recommendation quality. VIR-Bench thus establishes a critical foundation for advancing embodied AI and real-world navigation applications.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Recent advances in multimodal large language models (MLLMs) have significantly enhanced video understanding capabilities, opening new possibilities for practical applications. Yet current video benchmarks focus largely on indoor scenes or short-range outdoor activities, leaving the challenges associated with long-distance travel largely unexplored. Mastering extended geospatial-temporal trajectories is critical for next-generation MLLMs, underpinning real-world tasks such as embodied-AI planning and navigation. To bridge this gap, we present VIR-Bench, a novel benchmark consisting of 200 travel videos that frames itinerary reconstruction as a challenging task designed to evaluate and push forward MLLMs' geospatial-temporal intelligence. Experimental results reveal that state-of-the-art MLLMs, including proprietary ones, struggle to achieve high scores, underscoring the difficulty of handling videos that span extended spatial and temporal scales. Moreover, we conduct an in-depth case study in which we develop a prototype travel-planning agent that leverages the insights gained from VIR-Bench. The agent's markedly improved itinerary recommendations verify that our evaluation protocol not only benchmarks models effectively but also translates into concrete performance gains in user-facing applications.

Problem

• Evaluating MLLMs' geospatial-temporal understanding via travel itinerary reconstruction
• Addressing lack of benchmarks for long-distance travel video understanding
• Improving itinerary recommendations through enhanced geospatial-temporal intelligence

Innovation

• Introducing VIR-Bench for geospatial-temporal video evaluation
• Framing itinerary reconstruction as a core challenging task
• Developing a prototype agent for improved travel planning