Each Judge Its Own Yardstick: Discovering Per-VLM Taxonomies for Physical Video Evaluation

2026-06-22

AI Summary

This work addresses the limitation of existing video physical consistency evaluation frameworks, which apply a uniform standard and overlook the perceptual and representational disparities among vision-language models (VLMs), thereby failing to accurately reflect their true judgment capabilities. To overcome this, the authors propose JudgeFit, a method that constructs a customized evaluation taxonomy for each VLM by prompting it to enumerate physical inconsistencies, clustering these into initial dimensions, calibrating them with human commonsense ratings, and iteratively refining redundant or unreliable categories using a large language model until convergence. This approach enables, for the first time, model-specific assessment of physical reasoning. Evaluated across 16 VLMs, JudgeFit demonstrates an average 32% improvement in evaluation accuracy and effectively uncovers distinct blind spots and reliability patterns across different model families.

Abstract

Maintaining physical consistency in video generators and world models increasingly relies on vision-language models (VLMs) as automated judges that provide reward signals, ranking decisions, and data-filtering criteria. Yet VLMs differ substantially in training data and architecture, encoding physical phenomena through distinct internal representations. A single global evaluation schema therefore gives every VLM the same axes of competence, regardless of what each can actually perceive. We propose JudgeFit, an iterative refinement procedure that discovers a per-VLM evaluation taxonomy. An initial taxonomy is constructed by prompting the target VLM to enumerate physics errors on a small set of videos and clustering the resulting descriptions. The taxonomy is then refined through a diagnostic step: we calibrate the VLM's per-dimension scores to human physical-commonsense ratings, diagnose which dimensions it scores unreliably or redundantly, and prompt an LLM to repair them, iterating until convergence. We further instantiate this procedure as a benchmark and apply it to 16 VLMs spanning eight model families. The refined taxonomy outperforms the global-schema baseline on held-out videos for every VLM tested, with a mean relative improvement of approximately 32%. Beyond aggregate accuracy, the per-VLM profiles expose model-specific blind spots that overall rankings cannot anticipate, with reliability patterns differing markedly across model families.

Problem

• physical consistency
• vision-language models
• evaluation taxonomy
• video generation
• model-specific evaluation

Innovation

• per-VLM taxonomy
• JudgeFit
• physical consistency evaluation
• vision-language models
• iterative refinement