VIScore: Diagnosing Planning-Relevant Quality in Latent World Models

2026-08-11

AI Summary

Existing evaluation metrics struggle to effectively link latent space characteristics with planning success, particularly lacking diagnostic capability on out-of-distribution data. This work proposes VIScore, the first unified, quantifiable metric that jointly models the three stages of encoding, prediction, and search-based planning through three dimensions: Veracity, Influence, and Sobriety, comprehensively assessing a world model’s support for planning tasks. Experimental results demonstrate that VIScore achieves a Spearman correlation exceeding 0.75 across a cross-task success pool and exhibits significantly lower calibration error than constant fitting baselines. It attains state-of-the-art performance on both seen and unseen models and datasets, thereby transcending traditional evaluation paradigms that focus solely on latent representations.

Abstract

Regulating the latent space to an isotropic Gaussian distribution provides a stable and information-maximized landscape for world model planning. However, the latent space property and successful planning remain disconnected. We first study this by comparing SIGReg and VISReg, two regularization loss functions with the same distribution target but different properties. Compared with SIGReg, VISReg has more flexibility in controlling the weights of center, scale, and shape regularization, and a larger batch size brings a finer distribution approximation. We find that the former, despite being beneficial in self-supervised learning (SSL), does not help the planning, whereas the latter improves the planning success on out-of-domain (OOD) datasets. This motivates a deep understanding of the factors that correlate with the success rate. Unlike the previous metrics focusing on the encoded latent only, we propose the Veracity-Influence-Sobriety score (VIScore), a metric that quantifies the reachability and capacity of a predictor given the encoded feature, and the hallucination of the searching-based planner. Compared with straightness, physical-state probing, and empowerment, we show that, with the measurement covering encoder, predictor, and planner, VIScore explains the success rate better than the others, as reflected by a strong Spearman correlation. Specifically, VIScore consistently achieves a Spearman correlation over 0.75 on both seen and unseen models and datasets on the cross-task success rate pool. Moreover, VIScore is the only metric that has a calibration error below the constant fit across all testing scenarios, showcasing the importance of these three aspects in planning success. We hope this metric can help future studies on world model design and diagnosis.

Problem

• world models
• planning success
• latent space
• evaluation metrics
• out-of-domain generalization

Innovation

• VIScore
• latent world models
• planning-relevant quality
• regularization
• Spearman correlation