MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

2026-06-25

AI Summary

This work addresses a critical gap in the evaluation of video generation models: their inability to assess whether objects reappear correctly after occlusion, reflecting consistent memory of their state evolution during invisibility. To this end, the authors introduce MemoBench, a novel benchmark grounded in a “disappear–reappear” paradigm that systematically evaluates memory consistency under dynamic scene changes. MemoBench comprises 360 ground-truth video clips combining synthetic and real-world scenarios and features an automated evaluation metric alongside a four-dimensional visual question answering (VQA) diagnostic framework. Comprehensive assessment of eight state-of-the-art models reveals fundamental limitations in current approaches to modeling object memory under dynamic occlusion, thereby filling a significant void in existing evaluation methodologies.

Abstract

Video generation models aspire to simulate dynamic environments, and several benchmarks now evaluate memory consistency across frames. However, most assess consistency only while the target remains in view, and the few that force objects out of view evaluate static scenes where nothing changes during occlusion. To bridge this gap, we introduce MemoBench, a diagnostic benchmark built around the disappear-and-reappear paradigm in dynamically changing environments: a target object undergoes a physical process, disappears from view, and must be correctly recovered in its updated state upon reappearance. We curate 360 ground-truth clips spanning synthetic and real-world scenes, and design an evaluation suite combining automated metrics with VQA-based assessment across four diagnostic pillars. Evaluation of eight state-of-the-art models reveals key insights and open challenges regarding memory consistency under the disappear-and-reappear paradigm.

Problem

• memory consistency
• video generation
• dynamic environments
• occlusion
• world modeling

Innovation

• world modeling
• memory consistency
• disappear-and-reappear paradigm
• dynamic environments
• video generation benchmark