Distilling Game Code World Model Generation into Lightweight Large Language Models

2026-05-22

AI Summary

This work addresses the scalability limitations of existing game world models, which rely heavily on large language models and runtime optimization. To overcome this, the authors propose distilling the capability of generating executable code from game rules into a lightweight language model, enabling automatic construction of game environments—including rules, state transitions, and reward functions—directly from natural language descriptions. Leveraging a dataset of 30 games spanning both perfect and imperfect information settings, they introduce a post-training paradigm combining supervised fine-tuning (SFT) with reinforcement learning based on verifiable rewards (RLVR), augmented by dual structural and semantic validation mechanisms. Experiments on Qwen2.5-3B-Instruct demonstrate significant improvements in syntactic correctness and rule adherence of the generated code, efficiently producing valid GameCWMs across both game categories.

Abstract

Large Language Models (LLMs) have shown great ability in generating executable code from natural language, opening the possibility of automatically constructing environments for AI agents. Recent work on Code World Models (CWMs) demonstrates that LLMs can translate game rules into Python implementations compatible with solvers like Monte Carlo Tree Search. We study this problem in game settings, where generated environments must implement rules, legal actions, state transitions, observations, and rewards. We refer to these game-specific executable models as Game Code World Models (GameCWMs). However, current approaches to generating code world models rely on frontier models and inference-time refinement loops, limiting accessibility and scalability. This work investigates whether GameCWM generation capabilities can be distilled into smaller models through post-training. We introduce: (1) a curated dataset of 30 games spanning perfect and imperfect information games, (2) a verification framework that evaluates generated code against structural and semantic game properties, and (3) a post-training pipeline combining Supervised Fine-Tuning (SFT) with Reinforcement Learning with Verifiable Rewards (RLVR). We experiment with Qwen2.5-3B-Instruct and find that SFT can increase syntactic correctness, while RLVR can improve execution-level adherence to game rules, thereby improving Qwen's ability to generate valid GameCWMs in both perfect and imperfect information games. Overall, our pipeline makes Qwen2.5-3B-Instruct more capable of generating valid GameCWMs, thereby offering a scalable path toward automatic environment generation from natural language.

Problem

• Game Code World Models
• Large Language Models
• automatic environment generation
• scalability
• accessibility

Innovation

• Game Code World Models
• Knowledge Distillation
• Reinforcement Learning with Verifiable Rewards
• Supervised Fine-Tuning
• Executable Environment Generation