Language-conditioned world model improves policy generalization by reading environmental descriptions

2025-11-28

AI Summary

This work addresses the challenge of improving zero-shot policy generalization for language-conditioned world models (LC-WMs) in unseen game environments—without relying on test-time planning or expert demonstrations. To this end, we propose LED-WM: a novel LC-WM architecture that employs cross-modal attention to explicitly align natural language descriptions with visual-semantic entities in observations, and integrates language awareness into the world model encoder built upon DreamerV3. LED-WM generates synthetic language-conditioned trajectories to supervise end-to-end language-conditioned reinforcement learning and subsequent policy fine-tuning. Evaluated on the MESSENGER and MESSENGER-WM benchmarks, LED-WM achieves significant improvements in cross-task and cross-language zero-shot generalization. Crucially, it operates without any expert demonstrations or online planning. Our approach establishes a scalable, language-guided modeling paradigm for open-domain embodied intelligence.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

To interact effectively with humans in the real world, it is important for agents to understand language that describes the dynamics of the environment--that is, how the environment behaves--rather than just task instructions specifying "what to do". Understanding this dynamics-descriptive language is important for human-agent interaction and agent behavior. Recent work address this problem using a model-based approach: language is incorporated into a world model, which is then used to learn a behavior policy. However, these existing methods either do not demonstrate policy generalization to unseen games or rely on limiting assumptions. For instance, assuming that the latency induced by inference-time planning is tolerable for the target task or expert demonstrations are available. Expanding on this line of research, we focus on improving policy generalization from a language-conditioned world model while dropping these assumptions. We propose a model-based reinforcement learning approach, where a language-conditioned world model is trained through interaction with the environment, and a policy is learned from this model--without planning or expert demonstrations. Our method proposes Language-aware Encoder for Dreamer World Model (LED-WM) built on top of DreamerV3. LED-WM features an observation encoder that uses an attention mechanism to explicitly ground language descriptions to entities in the observation. We show that policies trained with LED-WM generalize more effectively to unseen games described by novel dynamics and language compared to other baselines in several settings in two environments: MESSENGER and MESSENGER-WM.To highlight how the policy can leverage the trained world model before real-world deployment, we demonstrate the policy can be improved through fine-tuning on synthetic test trajectories generated by the world model.

Problem

• Improves policy generalization using language-conditioned world models
• Eliminates reliance on planning or expert demonstrations
• Grounds language descriptions to observation entities via attention

Innovation

• Language-conditioned world model for policy generalization
• Attention mechanism grounds language to observation entities
• Model-based reinforcement learning without planning or expert demonstrations