Pixel2Phys: Distilling Governing Laws from Visual Dynamics

2026-02-23

AI Summary

This work addresses the challenge of automatically discovering low-dimensional, interpretable physical laws from high-dimensional visual data, which is hindered by pixel redundancy and information sparsity. The authors propose Pixel2Phys, a novel framework that integrates multi-agent collaboration with scientific reasoning to directly distill concise governing equations from raw video. For the first time, this approach combines iterative hypothesis generation, verification, and refinement within a pipeline compatible with any multimodal large language model. Crucially, it requires no predefined variables or simplifying assumptions, instead fusing symbolic regression with physics-informed constraint optimization. Evaluated across diverse simulated and real-world physical systems, Pixel2Phys accurately recovers underlying governing equations and demonstrates substantially superior long-term extrapolation performance compared to existing methods.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Discovering physical laws directly from high-dimensional visual data is a long-standing human pursuit but remains a formidable challenge for machines, representing a fundamental goal of scientific intelligence. This task is inherently difficult because physical knowledge is low-dimensional and structured, whereas raw video observations are high-dimensional and redundant, with most pixels carrying little or no physical meaning. Extracting concise, physically relevant variables from such noisy data remains a key obstacle. To address this, we propose Pixel2Phys, a collaborative multi-agent framework adaptable to any Multimodal Large Language Model (MLLM). It emulates human scientific reasoning by employing a structured workflow to extract formalized physical knowledge through iterative hypothesis generation, validation, and refinement. By repeatedly formulating, and refining candidate equations on high-dimensional data, it identifies the most concise representations that best capture the underlying physical evolution. This automated exploration mimics the iterative workflow of human scientists, enabling AI to reveal interpretable governing equations directly from raw observations. Across diverse simulated and real-world physics videos, Pixel2Phys discovers accurate, interpretable governing equations and maintaining stable long-term extrapolation where baselines rapidly diverge.

Problem

• physical laws
• visual dynamics
• high-dimensional data
• scientific discovery
• governing equations

Innovation

• scientific discovery
• physical law distillation
• multimodal reasoning
• interpretable AI
• video-to-dynamics