GameFactory: Creating New Games with Generative Interactive Videos

2025-01-14

AI Summary

Current generative game engines exhibit significant limitations in cross-scene generalization and precise action control, particularly when adapting to fixed-style games. To address this, we propose the first scene-generalization-driven framework for game video generation, built upon an open-source video diffusion model. Our method introduces a multi-stage fine-tuning strategy that decouples style learning from motion control, augmented by a novel motion-conditioning injection mechanism enabling arbitrarily long, action-controllable autoregressive video generation. We train and evaluate the framework on GF-Minecraft—a large-scale, real-world Minecraft dataset with fine-grained human action annotations. Experiments demonstrate substantial improvements in open-domain game video diversity, action fidelity, and cross-scene generalization capability. The code and dataset are publicly released, establishing a new paradigm for AI-driven game content generation.

Technology Category

• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Generative game engines have the potential to revolutionize game development by autonomously creating new content and reducing manual workload. However, existing video-based game generation methods fail to address the critical challenge of scene generalization, limiting their applicability to existing games with fixed styles and scenes. In this paper, we present GameFactory, a framework focused on exploring scene generalization in game video generation. To enable the creation of entirely new and diverse games, we leverage pre-trained video diffusion models trained on open-domain video data. To bridge the domain gap between open-domain priors and small-scale game dataset, we propose a multi-phase training strategy that decouples game style learning from action control, preserving open-domain generalization while achieving action controllability. Using Minecraft as our data source, we release GF-Minecraft, a high-quality and diversity action-annotated video dataset for research. Furthermore, we extend our framework to enable autoregressive action-controllable game video generation, allowing the production of unlimited-length interactive game videos. Experimental results demonstrate that GameFactory effectively generates open-domain, diverse, and action-controllable game videos, representing a significant step forward in AI-driven game generation. Our dataset and project page are publicly available at url{https://vvictoryuki.github.io/gamefactory/}.

Problem

• Generative Game Engines
• Video-to-Game Adaptation
• Multi-scene Compatibility

Innovation

• GameFactory
• Generative Game Engine
• AI Game Creation