Validate the Dream Before You Trust Its Verdict: Admissibility for World-Model Simulators

2026-07-08

AI Summary

This work addresses the lack of credible validation for action responses in existing world models (WMs) during policy evaluation, particularly under unseen actions. It proposes a tiered acceptability framework (L0–L4) for closed-loop verification, systematically integrating traditional simulation certification methodologies—including Verification, Validation & Accreditation (VV&A), Safety of the Intended Functionality (SOTIF), and scenario-based testing—into a hierarchical admission mechanism. Through action consistency tests, scenario replay, and joint evaluation using Fréchet Video Distance (FVD) and behavioral response metrics, empirical studies in autonomous driving reveal that high visual fidelity models (L0) exhibit significant deficiencies in action-following capability (L1–L2). This finding underscores a critical disconnect between visual quality and action robustness, demonstrating that visual metrics alone are insufficient for safety-critical validation.

Abstract

Across robotics, World Models (WMs) are increasingly used to evaluate action policies by simulating the consequences of actions in an imagined world, and returning a success or safety verdict. Yet a verdict is only as trustworthy as the WM that produced it, and the WM itself needs to be certified. In video-generation WMs, fidelity metrics such as Fréchet Video Distance (FVD) reward visual realism, but ignore whether the world responds correctly to the policy's actions, including those unseen in training. Classical simulation-based validation assumes a trusted simulator evaluating an untrusted policy, whereas generative WMs are themselves unverified learned artifacts. Hence, we argue that any WM used as a test oracle must first be accredited before its verdicts can serve as evidence. Building on credibility practices from safety-critical simulation, including Verification, Validation & Accreditation (VV&A), Safety of the Intended Functionality (SOTIF), and scenario-based testing standards, we define an admissibility ladder (L0-L4) that a WM must climb before its closed-loop verdicts are accepted as assurance evidence. Our framework is embodiment-agnostic, and is instantiated in autonomous driving (AD), where assurance methods for traditional simulation are most mature. Applied to two driving WMs, the lower rungs reveal a reversal: the model that ranks higher on visual generation quality (L0) ranks lower on action-following (L1-L2), so visual fidelity does not predict the action-robustness a closed-loop verdict depends on.

Problem

• World Models
• Admissibility
• Simulation Validation
• Action Robustness
• Assurance Evidence

Innovation

• World Models
• Admissibility Ladder
• Verification and Validation
• Closed-loop Evaluation
• Action Robustness