HarnessEval-W: Agentifying the Evaluation of Visual Worlds

2026-08-17

AI Summary

This study addresses the lack of verifiable reasoning and physical causal judgment in world model evaluation by proposing a novel hierarchical agent evaluation framework based on evidence trees. By integrating the Harness paradigm into the evaluation pipeline, this approach leverages multi-agent collaboration to decompose tasks and invoke specialized diagnostic tools, thereby constructing transparent evidence chains to justify scoring decisions. The method enables fine-grained, verifiable automated assessment. Extensive experiments across 18 models and 330 cases demonstrate that the framework’s judgments align closely with human evaluations, significantly enhancing assessment credibility. To facilitate reproducibility and further research, all relevant code and datasets have been made publicly available.

Abstract

A benchmark should deliver more than a scalar score: what makes an evaluation trustworthy is the reasoning that justifies the score. This is especially critical for world models, where judging a rollout requires understanding whether physics, causality, and world state evolve correctly. Humans spot such violations naturally, yet no existing benchmark automates this capability: metrics are computed brute-force, leaving no reasoning chain that can be examined or verified. We introduce HarnessEval-W, an agentified evaluation pipeline that brings the harness paradigm from the LLM ecosystem to world model benchmarking. Rather than applying a fixed rubric, HarnessEval-W interprets the context of each evaluation case, decomposes the evaluation question into measurable subproblems, and spawns specialized sub-agents, each equipped with tailored context and diagnostic tools to reason over its own subproblem. The parent agent then validates the gathered evidence and summarizes it into the final verdict. This hierarchical workflow turns every evaluation into a transparent evidence tree whose complete reasoning chain justifies the result. We apply HarnessEval-W to 18 representative world models over 330 evaluation cases. Its judgments closely align with human preferences while providing verifiable, fine-grained diagnoses of every generated rollout. We open-source the full pipeline as a live benchmark and invite the broad community to contribute to grow new skills and evaluation cases as world models evolve.

Problem

• World Model Evaluation
• Reasoning Chain
• Benchmark Trustworthiness
• Visual Worlds

Innovation

• Agentified Evaluation
• World Models
• Hierarchical Workflow
• Explainable Benchmark
• Evidence Tree