Physics-IQ Verified

2026-06-17

AI Summary

Current evaluations of physical understanding in video generation models suffer from biases that hinder accurate assessment of their grasp of real-world physical laws. To address this, this work presents a systematic reconstruction of the Physics-IQ benchmark by enhancing prompt design and real-video quality, eliminating confounding factors, and introducing an equal-weight, sample-level scoring mechanism. The revised benchmark replaces 57.6% of samples and 34.8% of prompts, leading to substantial shifts in model rankings—evidenced by a Kendall’s τ of 0.46 across six image-to-video models—and markedly improves the fairness, reliability, and validity of physical reasoning evaluation.

Abstract

Video generative models ( VGMs) have become a new frontier that can be used not just for video generation but for a multitude of downstream tasks, including world modeling. To advance these tasks, a good video model must understand the physical reality of the world. Evaluating this understanding is an emerging field and has led to the Physics-IQ benchmark, which quantifies this explicitly by comparing model-generated videos to real-world videos of physical experiments. In this work, we present a systematic audit of the Physics-IQ benchmark, expose shortcomings and propose three solutions that sharpen how we can measure physical understanding of VGMs. Specifically, we improve prompt and ground-truth quality to reduce the influence of confounding factors and further introduce a sample-level scoring system that weights each sample and metric equally. Our resulting benchmark, Physics-IQ Verified, refines 57.6\% of all samples and improves over 34.8\% of prompts. In a comparison study using six image-to-video generative models, we observe moderate but meaningful ranking changes (Kendall's $τ= 0.46$). We hope Physics-IQ Verified advances the community by providing a more reliable signal toward physically accurate VGMs. The code for the benchmark can be accessed at https://github.com/google-deepmind/physics-iq-benchmark

Problem

• video generative models
• physical understanding
• benchmark evaluation
• Physics-IQ
• world modeling

Innovation

• Physics-IQ Verified
• video generative models
• physical understanding
• benchmark refinement
• sample-level scoring