SimuScene: Training and Benchmarking Code Generation to Simulate Physical Scenarios

2026-02-11

AI Summary

This work addresses the lack of systematic research and evaluation benchmarks for large language models (LLMs) in code-based modeling and simulation of physical scenarios. We present the first comprehensive code generation benchmark spanning five physics domains and 52 core concepts, supported by a high-quality dataset of 7,659 curated scenarios—including 334 expert-validated test samples—constructed through automated collection and manual verification. To enhance both physical plausibility and general code generation capabilities, we propose a reinforcement learning training paradigm that employs a vision-language model as a discriminator and incorporates a visual reward mechanism. Our evaluation of ten prominent LLMs reveals that even the strongest baseline achieves only a 21.5% pass rate, whereas our approach significantly improves performance in both physical consistency and code correctness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large language models (LLMs) have been extensively studied for tasks like math competitions, complex coding, and scientific reasoning, yet their ability to accurately represent and simulate physical scenarios via code remains underexplored. We propose SimuScene, the first systematic study that trains and evaluates LLMs on simulating physical scenarios across five physics domains and 52 physical concepts. We build an automatic pipeline to collect data, with human verification to ensure quality. The final dataset contains 7,659 physical scenarios with 334 human-verified examples as the test set. We evaluated 10 contemporary LLMs and found that even the strongest model achieves only a 21.5% pass rate, demonstrating the difficulty of the task. Finally, we introduce a reinforcement learning pipeline with visual rewards that uses a vision-language model as a judge to train textual models. Experiments show that training with our data improves physical simulation via code while substantially enhancing general code generation performance.

Problem

• code generation
• physical simulation
• large language models
• physics reasoning
• scenario modeling

Innovation

• physical simulation
• code generation
• large language models
• reinforcement learning
• vision-language model