DynaVieW: Schema-Guided World Modeling for Understanding Hierarchical Visual Dynamics

2026-07-05

AI Summary

Existing multimodal large language models struggle to systematically model the temporal evolution of hierarchical visual dynamics—such as actions and their consequent environmental changes—in video or multi-image sequences. To address this limitation, this work proposes DynaVieW, a dynamic pattern-guided world model that constructs state-transition sequences from keyframes and introduces hierarchical dynamic patterns to jointly optimize state simulation and transition prediction. The approach integrates a mixture-of-experts architecture, cross-expert selective attention mechanisms, and a pattern-token reweighting loss to enable structured, controllable, and robust modeling of complex visual dynamics. Experimental results demonstrate that DynaVieW significantly improves consistency, controllability, and instruction-following capabilities in visual storytelling and world simulation tasks, validating its effectiveness in understanding and generating multilevel visual dynamics.

Abstract

Multimodal LLMs struggle to systematically model the temporal evolution of visual scenes in videos or multi-image sequences. Such inputs require models to predict or simulate multiple levels of dynamic constituents, such as actions taken in the visual sequence, and the associated changes to the visual environment that result. To address this challenge, we propose a dynamic schema-guided world model, DynaVieW, optimized for visual dynamic prediction and simulation. DynaVieW achieves an in-depth understanding of visual dynamics by learning interleaved state-transition sequences, where states cover broad visual scenes from video keyframes, and transitions capture comprehensive dynamic constituents within a hierarchical schema. DynaVieW jointly models transition prediction and state simulation under a mixture-of-experts architecture, with a cross-expert selective attention and a schema token re-weighted loss, to ensure effective and robust learning. DynaVieW's understanding of visual dynamics boosts its downstream performance in visual narrative creation and world simulation, showing improved consistency, controllability, and instruction-following.

Problem

• visual dynamics
• temporal evolution
• multimodal LLMs
• hierarchical modeling
• world modeling

Innovation

• schema-guided world modeling
• hierarchical visual dynamics
• mixture-of-experts
• state-transition sequences
• visual narrative generation