Marionette: Predicting World States, Rendering Geometry, Painting Appearance

2026-08-14

AI Summary

This study addresses the challenges of poor temporal consistency and limited controllability in long-horizon generation for interactive game world models. We propose a decoupled architecture that separates state prediction from appearance synthesis by integrating explicit 3D state modeling with a zero-parameter renderer for precise geometric computation, alongside a video diffusion model for visual rendering. This framework enables interpretable interactive character generation and supports direct behavioral correction within an explicit state space. Experimental results demonstrate that motion intervention reduces joint error by 31%, while state-based constraints decrease ground penetration by 66%, achieving an FVD score of 831. These findings indicate significant improvements in generation fidelity, long-term temporal consistency, and controllability for interactive world modeling.

Abstract

Interactive game world models typically autoregress visual observations directly in pixel or latent space, forcing structured properties such as pose, geometry, and occlusion to be implicitly maintained by the same generative sequence. Over long horizons, errors in these latent world properties accumulate, making consistency and controllability fragile. We explicitly model the evolving world state, delegate exact geometric computation to a fixed, zero-parameter renderer, and leave the neural model to synthesize appearance. We instantiate this idea as Marionette, a world model for interactive games with articulated characters. First, a two-stage autoregressive dynamics model predicts an explicit and interpretable 276-dimensional 3D world state comprising multi-entity articulated skeletons, metric root trajectories, and rotations. Second, a zero-parameter graphics bridge converts the predicted state into pose-control videos, computing world-space geometry and occlusion in closed form. Third, a control-conditioned video-diffusion observation model synthesizes photorealistic RGB observations from the resulting structured controls. Our experiments establish two properties of Marionette. First, the predicted world state is directly controllable. Forcing a mismatched action stream changes root-aligned joint error by 31% across 48 held-out segments. Second, long-horizon behaviour is determined in the state, and can be repaired there. Left free, the two generated characters drift to 21.2 m apart (recorded sessions stay near 5 m) and a third of frames show ground penetration. Two rules imposed on the explicit state, a terrain collider and a separation cap, cut penetration by 66% and keep the pair engaged, with no change to the observation model. Routing appearance through the predicted state costs no fidelity we can detect, at an FVD of 831 against 799 for recorded pose.

Problem

• Interactive game world models
• Long-horizon consistency
• Controllability
• Error accumulation
• Implicit latent representation

Innovation

• Explicit World State
• Zero-parameter Renderer
• Decoupled Generation
• Video Diffusion
• Long-horizon Consistency