Semantic World Models

2025-10-22

AI Summary

Traditional world models rely on pixel-level future frame prediction; however, strong reconstruction capability does not necessarily correlate with effective planning performance, leading to poor generalization. This work proposes a Semantic World Model (SWM), reframing world modeling as a task-oriented visual question answering (VQA) problem: instead of reconstructing pixels, it directly predicts task-relevant semantic information—such as object states and spatial relations—in future frames conditioned on actions. Methodologically, we introduce vision-language models (VLMs) into the world model framework for the first time, training them via supervised fine-tuning on image-action-text triplets to enable action-conditional semantic forecasting and end-to-end decision-making. Experiments demonstrate that our approach significantly improves policy generalization across open-ended robotic tasks, outperforming state-of-the-art pixel-reconstruction-based world models.

Technology Category

• Computer Vision: Large Vision Models
• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Planning with world models offers a powerful paradigm for robotic control. Conventional approaches train a model to predict future frames conditioned on current frames and actions, which can then be used for planning. However, the objective of predicting future pixels is often at odds with the actual planning objective; strong pixel reconstruction does not always correlate with good planning decisions. This paper posits that instead of reconstructing future frames as pixels, world models only need to predict task-relevant semantic information about the future. For such prediction the paper poses world modeling as a visual question answering problem about semantic information in future frames. This perspective allows world modeling to be approached with the same tools underlying vision language models. Thus vision language models can be trained as"semantic"world models through a supervised finetuning process on image-action-text data, enabling planning for decision-making while inheriting many of the generalization and robustness properties from the pretrained vision-language models. The paper demonstrates how such a semantic world model can be used for policy improvement on open-ended robotics tasks, leading to significant generalization improvements over typical paradigms of reconstruction-based action-conditional world modeling. Website available at https://weirdlabuw.github.io/swm.

Problem

• Predicting task-relevant semantic information for planning
• Replacing pixel reconstruction with visual question answering
• Enabling robust generalization in robotic decision-making

Innovation

• Predicts task-relevant semantic future information
• Models world as visual question answering problem
• Uses vision language models for semantic world modeling