Sekai: A Video Dataset towards World Exploration

2025-06-18

AI Summary

Existing video generation datasets suffer from narrow geographic coverage, short durations, static scenes, and insufficient semantic annotations—limiting their utility for world exploration modeling. To address this, we introduce Sekai, the first high-quality, first-person video dataset explicitly designed for interactive world exploration: it spans 750 cities across 100+ countries and comprises over 5,000 hours of walking and aerial footage. We develop an end-to-end pipeline for acquisition, preprocessing, and annotation—enabling automated geocoding, multimodal semantic labeling (e.g., location, weather, crowd density, descriptive captions, camera trajectory), and spatiotemporally aligned fine-grained scene understanding. Sekai overcomes four key bottlenecks—geographic breadth, temporal length, scene dynamics, and semantic richness—thereby enabling the training of YUME, a novel exploration model that significantly improves spatial consistency and exploration plausibility in generated outputs.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Video generation techniques have made remarkable progress, promising to be the foundation of interactive world exploration. However, existing video generation datasets are not well-suited for world exploration training as they suffer from some limitations: limited locations, short duration, static scenes, and a lack of annotations about exploration and the world. In this paper, we introduce Sekai (meaning ``world'' in Japanese), a high-quality first-person view worldwide video dataset with rich annotations for world exploration. It consists of over 5,000 hours of walking or drone view (FPV and UVA) videos from over 100 countries and regions across 750 cities. We develop an efficient and effective toolbox to collect, pre-process and annotate videos with location, scene, weather, crowd density, captions, and camera trajectories. Experiments demonstrate the quality of the dataset. And, we use a subset to train an interactive video world exploration model, named YUME (meaning ``dream'' in Japanese). We believe Sekai will benefit the area of video generation and world exploration, and motivate valuable applications.

Problem

• Existing video datasets lack diversity for world exploration training
• Current datasets have limited locations, duration, and annotations
• Need high-quality annotated videos for interactive world exploration models

Innovation

• High-quality first-person worldwide video dataset
• Efficient toolbox for video collection and annotation
• Interactive video world exploration model YUME