World Models as an Intermediary between Agents and the Real World

2026-01-31

AI Summary

This work addresses the challenges of low sample efficiency and prolonged training cycles faced by reinforcement learning agents in high-cost, complex domains such as robotics, scientific experimentation, and machine learning engineering, where environmental interactions are expensive. It presents the first systematic demonstration of the pivotal role of world models in cross-domain, high-cost tasks: by modeling environment dynamics, reward mechanisms, and task distributions, world models serve as intermediaries to real-world interaction, providing rich learning signals that mitigate severe off-policy issues. The study identifies key challenges in data curation, architectural design, scalability, and evaluation, and validates across multiple domains that world models substantially enhance both learning efficiency and agent performance.

Technology Category

• Machine Learning: Reinforcement Learning
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Multiagent Learning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Large language model (LLM) agents trained using reinforcement learning has achieved superhuman performance in low-cost environments like games, mathematics, and coding. However, these successes have not translated to complex domains where the cost of interaction is high, such as the physical cost of running robots, the time cost of ML engineering, and the resource cost of scientific experiments. The true bottleneck for achieving the next level of agent performance for these complex and high-cost domains lies in the expense of executing actions to acquire reward signals. To address this gap, this paper argues that we should use world models as an intermediary between agents and the real world. We discuss how world models, viewed as models of dynamics, rewards, and task distributions, can overcome fundamental barriers of high-cost actions such as extreme off-policy learning and sample inefficiency in long-horizon tasks. Moreover, we demonstrate how world models can provide critical and rich learning signals to agents across a broad set of domains, including machine learning engineering, computer use, robotics, and AI for science. Lastly, we identify the challenges of building these world models and propose actionable items along dataset curation, architecture design, scaling, and evaluation of world models.

Problem

• world models
• high-cost interaction
• reinforcement learning agents
• sample inefficiency
• complex domains

Innovation

• world models
• reinforcement learning
• sample efficiency
• high-cost domains
• off-policy learning