GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?

2026-06-16

AI Summary

This work addresses the challenge of evaluating agents capable of end-to-end generation of playable, complete, and interaction-coherent games from natural language instructions within a realistic game engine. To this end, we introduce GameCraft-Bench, an evaluation framework built on the Godot engine that systematically encompasses 15 game genres and 140 tasks. We propose a multimodal evaluation paradigm grounded in engine fidelity, content completeness, and interaction consistency, integrating replayable demonstrations with rule-guided scoring. Experimental results reveal that even the strongest current agents achieve only a 41.46% success rate; while many models can implement basic mechanics, they exhibit significant deficiencies in content completeness, visual feedback, and overall presentation quality.

Abstract

Game generation is an emerging application of coding agents, requiring models to transform natural-language specifications into playable interactive systems. Unlike traditional coding tasks, game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay. We formalize end-to-end game generation as the problem of producing a complete game artifact that realizes a specification through observable player-game interaction in a target environment. We argue that evaluating this setting requires three desiderata: Engine Grounding, Artifact Completeness, and Interactive Verification. We propose an interaction-grounded evaluation framework that assesses executable gameplay through replayed demonstrations and rubric-guided multimodal judging. We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families. Evaluations of frontier coding agents show that end-to-end game generation remains highly challenging: the strongest agent achieves only 41.46%, and most agents score below 40%. Further analysis reveals that while agents often implement recognizable mechanics, they struggle to deliver complete games with sufficient content, functional visual feedback, and coherent presentation. See https://tongxuluo.github.io/gamecraft-bench-website for demos, code, and data.

Problem

• game generation
• coding agents
• game engine
• end-to-end generation
• interactive systems

Innovation

• end-to-end game generation
• game engine grounding
• interactive verification
• artifact completeness
• multimodal evaluation