Gravity-Bench-v1: A Benchmark on Gravitational Physics Discovery for Agents

2025-01-30

AI Summary

This work addresses the evaluation of AI systems’ ability to discover physical laws—particularly gravitational laws—and generalize to non-standard physical environments. To this end, we introduce the first AI agent benchmark specifically designed for gravitational physics discovery, requiring agents to actively sample observations, perform online analysis, and infer hidden dynamical laws within a simulation under limited experimental budgets, covering both realistic and counterfactual gravitational scenarios. Our methodology integrates high-fidelity gravitational simulation, active experimental design, dynamic causal modeling, and symbolic induction. Key contributions include: (1) the first open-solution-space benchmark explicitly oriented toward scientific discovery; (2) out-of-distribution (OOD) physical perturbations to assess genuine scientific generalization; and (3) a PhD-level reference solution serving as a human expert benchmark. Experiments show that our benchmark significantly surpasses existing AI baselines at an advanced undergraduate level, establishing a quantifiable, scalable evaluation paradigm for AI-driven scientific discovery.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Modern science emerged from reasoning over repeatedly-observed planetary motions. We present Gravity-Bench-v1, an environment-based benchmark that challenges AI agents on tasks that parallel this historical development. Gravity-Bench-v1 evaluates agents on the discovery of physics concealed within a dynamic environment, using rigorous gravitational dynamics simulations. Gravity-Bench includes out-of-distribution cases, i.e. with physics that deviates from the real world, to evaluate true scientific generalization capabilities. Agents must plan to collect data within an experimental budget and must perform a dynamic form of data analysis and reasoning to solve tasks efficiently. Our benchmark admits an open-ended space of solutions. PhD-level solutions for each task are provided, to calibrate AI performance against human expertise. Technically at an upper-undergraduate level, our benchmark proves challenging to baseline AI agents. Gravity-Bench-v1 and planned extensions should help map out AI progress towards scientific discovery capabilities.

Problem

• AI systems
• fundamental physical laws
• generalization in non-standard environments

Innovation

• AI Physics Understanding
• Gravitational Law Simulation
• Scientific Discovery Assessment