In-Context World Modeling for Robotic Control

2026-06-24

AI Summary

Existing vision-language-action (VLA) models struggle to generalize to novel environments—such as those with different camera viewpoints or robot configurations—because they rely solely on current observations and instructions while ignoring dynamic changes in system configuration. This work proposes an in-context world modeling framework that, for the first time, formulates system identification as an in-context adaptation problem: by leveraging task-agnostic, short-horizon interaction histories, the method implicitly infers environmental dynamics and system variables, enabling context-aware adaptation without any parameter updates. Integrating in-context learning, self-generated interaction data, and VLA models, the approach yields a fine-tuning-free dynamic control policy. Experiments demonstrate significant improvements over standard VLA baselines in both simulation and real-world robotic platforms, with particularly strong generalization under novel camera viewpoints.

Abstract

Modern Vision-Language-Action (VLA) models often fail to generalize to novel setups, such as altered camera viewpoints or robot morphologies, because they are typically conditioned only on current observations and language instructions. By ignoring the underlying system configuration as a variable, these models implicitly assume a fixed execution context encountered during training, necessitating data-intensive fine-tuning for any new environment. In this work, we introduce In-Context World Modeling (ICWM), a framework that treats system identification as an in-context adaptation problem. ICWM enables robot policies to autonomously infer essential system variables from a short history of self-generated, task-agnostic interactions. Unlike traditional In-Context Learning that uses demonstrations to specify what task to perform, ICWM leverages the context window to understand how the system operates. By processing these interactions before task execution, the model implicitly captures the world dynamics of the current system, enabling adaptation to novel configurations without parameter updates. Extensive experiments in simulation and on real-world robot platforms demonstrate that ICWM significantly outperforms standard VLA baselines on novel camera viewpoints.

Problem

• Vision-Language-Action models
• generalization
• system configuration
• novel environments
• robotic control

Innovation

• In-Context Learning
• World Modeling
• System Identification
• Robotic Generalization
• Vision-Language-Action Models