RetroHolmes: When Semantic Plausibility Fails Retrospective Physical Process Reasoning

2026-07-12

AI Summary

Current evaluations of vision-language models lack effective tests for retrodictive physical reasoning—the ability to infer past physical causes from observed outcomes. This work proposes a novel evaluation paradigm by introducing RetroHolmes, the first real-world benchmark specifically designed for retrodictive physical reasoning. RetroHolmes comprises object-centric image pairs, annotated causal step sequences, and reachability labels, and is paired with a vision-simulation-driven analysis-by-synthesis reasoning framework. Experiments reveal that prevailing models exhibit systematic biases in reachability judgments and frequently disregard physical evidence, underscoring the diagnostic value of RetroHolmes. Furthermore, the results demonstrate that physically grounded intermediate representations are crucial for enhancing retrodictive reasoning capabilities.

Abstract

Humans can infer hidden physical processes from sparse observations, yet current evaluation protocols for Vision Language Models fail to assess whether such physical reasoning is genuinely captured. To address this gap, we introduce Retrospective Physical Process Reasoning, a new evaluation paradigm to reason backward from outcomes under explicit physical constraints. Building on the paradigm, we present RetroHolmes, the first real-world benchmark for Retrospective Physical Process Reasoning, comprising object-centric image pairs annotated with reachability labels and causal step sequences across diverse physical transitions. Using RetroHolmes, we analyze state of the art Vision Language Models and uncover systematic failure modes, including judgment bias in reachability assessment and belief dominance over physical evidence, mirroring sycophancy behavior observed in large language models. We further demonstrate a simple analysis-by-synthesis instantiation with visual simulation as an intermediate step, validating the diagnostic value of RetroHolmes and highlighting the importance of physically grounded intermediate representations for physical reasoning.

Problem

• Retrospective Physical Process Reasoning
• Vision Language Models
• physical reasoning
• evaluation benchmark
• hidden physical processes

Innovation

• Retrospective Physical Process Reasoning
• Vision Language Models
• Physical Reasoning
• RetroHolmes
• Analysis-by-Synthesis