Dreaming in Code for Curriculum Learning in Open-Ended Worlds

2026-02-09

AI Summary

In open-world settings, agents struggle to construct continuously learnable experience sequences due to the vast combinatorial space of tasks, hindering long-term skill acquisition. To address this challenge, this work proposes the Dreaming in Code (DiCode) framework, which introduces code-level environment generation into curriculum learning for the first time. DiCode leverages foundation models to dynamically synthesize executable environment code, creating procedural “dream-like” variants that form a progressive curriculum to bridge capability gaps. Evaluated on the Craftax benchmark, DiCode substantially improves agent performance on long-horizon tasks, achieving a 16% increase in average return and, notably, the first non-zero success rate on late-stage combat tasks where prior methods completely fail.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Open-ended learning frames intelligence as emerging from continual interaction with an ever-expanding space of environments. While recent advances have utilized foundation models to programmatically generate diverse environments, these approaches often focus on discovering isolated behaviors rather than orchestrating sustained progression. In complex open-ended worlds, the large combinatorial space of possible challenges makes it difficult for agents to discover sequences of experiences that remain consistently learnable. To address this, we propose Dreaming in Code (DiCode), a framework in which foundation models synthesize executable environment code to scaffold learning toward increasing competence. In DiCode,"dreaming"takes the form of materializing code-level variations of the world. We instantiate DiCode in Craftax, a challenging open-ended benchmark characterized by rich mechanics and long-horizon progression. Empirically, DiCode enables agents to acquire long-horizon skills, achieving a $16\%$ improvement in mean return over the strongest baseline and non-zero success on late-game combat tasks where prior methods fail. Our results suggest that code-level environment design provides a practical mechanism for curriculum control, enabling the construction of intermediate environments that bridge competence gaps in open-ended worlds. Project page and source code are available at https://konstantinosmitsides.github.io/dreaming-in-code and https://github.com/konstantinosmitsides/dreaming-in-code.

Problem

• open-ended learning
• curriculum learning
• environment generation
• long-horizon skills
• combinatorial challenge space

Innovation

• Dreaming in Code
• curriculum learning
• open-ended environments
• code-level environment generation
• foundation models