Reference-Free Assessment of Physical Consistency in World Model-based Video Generation

2026-06-21

AI Summary

Existing video generation evaluation methods rely on human ratings or ground-truth reference videos, making it difficult to effectively assess the physical consistency of videos generated by world models and resulting in a significant performance gap when transferring from simulation to real-world tasks. This work proposes the first reference-free automatic evaluation method for physical consistency, integrating both relative and absolute assessment strategies. By leveraging DROID-SLAM and SEA-RAFT, the approach constructs a spatiotemporal consistency metric that precisely localizes the timing and spatial location of physical artifacts. Experimental results demonstrate that videos selected using this method improve downstream task success rates by over 8%, substantially narrowing the sim-to-real performance gap.

Abstract

We introduce reference-free measures for evaluating the physical consistency of generated videos, combining relative and absolute approaches to assess fidelity. Although tools like WorldGym or WorldEval enable robotic simulation via video generation, physical fidelity gaps often prevent these environments from accurately reproducing real-world task success rates of VLA models. Unlike existing evaluation methods, which require costly human voting (Elo) or unavailable ground-truth references (FVD), our approach utilizes DROID-SLAM and SEA-RAFT to quantify physical inconsistencies, motivated by WorldScore. Videos filtered using our relative consistency assessment show an improvement in task success rates of over 8%, effectively narrowing the simulation-to-reality gap. Furthermore, our absolute assessment enables spatio-temporal localization, providing visualization of when and where physical artifacts occur.

Problem

• physical consistency
• video generation
• world models
• reference-free evaluation
• simulation-to-reality gap

Innovation

• reference-free evaluation
• physical consistency
• video generation
• world models
• spatio-temporal localization