Multiplayer Interactive World Models with Representation Autoencoders

2026-07-06

AI Summary

Existing single-agent world models struggle to capture the distinct influence of each agent’s actions on scene evolution in multi-player dynamic environments. This work proposes the first world model capable of supporting multi-player interaction by explicitly conditioning on multiple agents’ action streams, enabling long-term consistent scene prediction in highly dynamic physical settings. Built upon a 5-billion-parameter latent diffusion architecture, the model integrates autoencoded representations, a dedicated video codec, and a multi-player conditioning mechanism, trained and validated in Rocket League. It generates four-player gameplay videos at 20 FPS in real time on a single NVIDIA B200 GPU, maintaining simulation stability beyond five minutes—with some scenarios remaining coherent for several hours—significantly advancing physical consistency and temporal stability in multi-agent world modeling.

Abstract

We introduce the first multiplayer world model for highly dynamic environments governed by complex physical interactions. Whereas single-player world models treat the other agents as part of the environment, ours conditions on the action streams of multiple agents, learning to attribute changes in the scene to the correct player and to stay coherent under arbitrary combinations of their actions. We study this problem in the game of Rocket League, where players compete and cooperate under fast, tightly coupled dynamics. Trained on 10,000 hours of gameplay collected with publicly available bots, our 5-billion-parameter latent diffusion model generates four-player matches in real time, producing 20 frames per second on a single Nvidia B200 GPU. Although trained only on short clips, its rollouts stay stable far beyond the training horizon: distributional quality holds steady out to five minutes, the longest horizon we measure, and in practice we observe rollouts continuing for hours with no sign of collapse. We systematically investigate the central design choices: the video codec, the generative objective, and the multiplayer conditioning scheme. In addition, we characterize how behavior changes with model and data scale, including the capabilities that emerge and the failure modes that persist. We further develop targeted evaluations that probe the model's physical understanding rather than visual appearance alone. To support continued research on multiplayer world models, we release our dataset, our full training and inference codebase, and a live demo.

Problem

• multiplayer world models
• complex physical interactions
• action attribution
• dynamic environments
• coherent prediction

Innovation

• multiplayer world models
• representation autoencoders
• latent diffusion models
• physical interaction modeling
• long-horizon rollouts