EgoCS-400K: An Egocentric Gameplay Dataset for World Models

2026-06-16

AI Summary

This work addresses a critical limitation in current world model research—the scarcity of large-scale, temporally aligned multimodal interaction data—by introducing the first large-scale first-person dataset derived from professional Counter-Strike match replays. By parsing game replays and rendering player viewpoints, the dataset automatically extracts and precisely aligns multimodal trajectories, including video, actions, agent states, camera views, and in-game events. Spanning 13 maps, 40,000 rounds, and over 400,000 video clips (approximately 10,000 hours in total), it achieves a strong balance among scene diversity, action realism, and data scale. The dataset enables interactive visual modeling tasks such as action-conditioned video prediction and state-aware scene forecasting, offering a rich benchmark for advancing embodied AI and world modeling in complex, dynamic environments.

Abstract

The shift from video generation to interactive world modeling places new demands on data: beyond captioned videos, world models require temporally aligned video-action-language trajectories grounded in the actions, camera motion, states, and events that drive future scene changes. However, such data is difficult to obtain at scale. Web video datasets offer broad visual coverage but lack executable actions and reliable states; robotic datasets provide action and state supervision but are costly and limited in scene diversity; and existing simulators often lack large-scale human-driven interaction trajectories. In this paper, we introduce EgoCS-400K, a large-scale replay-grounded egocentric Counter-Strike dataset for world models, built from public professional CS and CS2 match demos that preserve human gameplay trajectories and enable parsing, replaying, rendering, and temporal alignment. We extract player states, view directions, movements, keyboard/button inputs, view-angle changes, weapon usage, game events, and round-level context, and render clean first-person videos from the same trajectories. EgoCS-400K contains over 400,000 first-person videos and 10,000 hours of gameplay from more than 1,000 matches and 40,000 rounds, covering 13 maps and 10 player viewpoints per round. It supports a range of interactive visual modeling tasks, including action-conditioned future prediction, state- and event-aware scene rollout, replay-grounded captioning, and agent egocentric action understanding. By connecting visual observations with human actions, camera motion, game states, and events at scale, EgoCS-400K serves as a practical bridge between passive web videos, controllable game simulation, and costly real-world embodied data.

Problem

• world models
• egocentric data
• action-state alignment
• interactive trajectories
• scalable dataset

Innovation

• egocentric dataset
• world models
• gameplay trajectories
• temporal alignment
• interactive simulation