NICE FACT: Diagnosing and Calibrating VLMs in Quantitative Reasoning for Kinematic Physics

2026-05-08

AI Summary

This work addresses the limited performance of current vision-language models (VLMs) on spatially grounded tasks such as kinematics and the absence of fine-grained diagnostic tools for evaluating their physical reasoning capabilities. The authors propose a dual-module diagnostic framework, NICE and FACT: FACT decomposes quantitative reasoning to assess model proficiency across three dimensions—visual fidelity, understanding of physical laws, and temporal alignment—while NICE introduces a neighborhood-aware mechanism to calibrate model confidence. This framework establishes the first standardized benchmark for physics-oriented reasoning in VLMs. Experiments on six state-of-the-art models reveal pervasive issues, including misidentification of visual premises and incorrect application of physical principles. The proposed approach effectively uncovers these shortcomings and enhances the reliability of model confidence estimates.

Abstract

The ability to derive precise spatial and physical insights is a cornerstone of vision-language models (VLMs), yet their poor performances in related spatial intelligence tasks such as physical reasoning remain a fundamental barrier. The community critically lacks a scientific analysis revealing whether VLMs faithfully reach answers or plausibly make guesses. This work aims to provide a fundamental understanding of how VLMs perceive the physical world, and utilize physical laws, while assessing the reliability of model confidence. We propose NICE and FACT, a dual-diagnostic paradigm that explicitly decomposes quantitative reasoning for kinematic physics: FACT diagnoses visual fidelity, physical law comprehension, and temporal grounding. NICE studies our novel neighborhood-informed calibration method and novel metrics to evaluate and calibrate confidence reliability. Evaluated across 6 latest state-of-the-art VLMs, we uncover that models fail to identify visual preconditions or utilize necessary physical laws to reach answers. This work highlights and establishes a standardized diagnostic paradigm to guide the development of faithful, physically-grounded VLMs.

Problem

• vision-language models
• quantitative reasoning
• kinematic physics
• physical reasoning
• model calibration

Innovation

• quantitative reasoning
• kinematic physics
• visual-language models
• confidence calibration
• diagnostic paradigm