Reinforcement Learning with Verifiable Physics: Post-training LLMs with Continuous Rewards

2026-07-11

AI Summary

This work addresses the limitations of traditional PDE solvers, which rely heavily on expert knowledge and laborious development, as well as existing large language model (LLM) approaches that focus primarily on reasoning optimization while lacking fine-grained feedback on scientific computation accuracy. The authors propose RLVP, a novel framework that introduces, for the first time, a physics-consistency-based continuous reward mechanism combined with a hard constraint on program executability to train LLMs via reinforcement learning for generating high-accuracy solver code. This approach overcomes the shortcomings of conventional binary verification in scientific computing, significantly outperforming both pretrained and supervised fine-tuning baselines across multiple PDE benchmarks. Notably, even smaller models trained with RLVP surpass state-of-the-art prompting strategies of larger models and demonstrate strong zero-shot transfer across PDE types and compositional generalization of numerical modules.

Abstract

Partial differential equations (PDEs) are foundational to modeling in science and engineering, but constructing reliable numerical solvers remains labor-intensive, demanding expert knowledge of discretization schemes, stability conditions, and boundary treatments. Recent work has begun to frame PDE solving as a code-generation task for large language models (LLMs), yet existing approaches operate primarily at inference time: relying on prompting, debugging, self-refinement, and test-time scaling rather than adapting the model itself. In parallel, reinforcement learning with verifiable rewards has emerged as a post-training paradigm for code and math reasoning, but its verifiers are typically binary: a compiler runs, or a test passes. Such signals discard the graded structure of scientific correctness, where two solvers may both execute and yet differ in solution accuracy by orders of magnitude. In this work, we introduce RLVP: Reinforcement Learning with Verifiable Physics, an RL post-training framework for multi-PDE solver code generation. RLVP addresses this verifiability gap with a hybrid verifier: hard program-validity checks ensure executability, while continuous physics rewards score function-space accuracy and PDE-residual consistency. A single policy is post-trained across diverse PDE families spanning hyperbolic, parabolic, elliptic, and incompressible-flow systems. RLVP improves over both pre-trained and supervised-only baselines on PDE benchmarks, and shows zero-shot improvement transfer to held-out PDEs. We show that a smaller LLM post-trained with RLVP can outperform prompting a frontier model on in-distribution PDE solver generation. The trained policy shows evidence of compositionality in numerical motifs: it recombines stencils, time-stepping schemes, and boundary-handling primitives learned from the PDEs used in training into generated solvers for unseen PDE problems.

Problem

• Partial Differential Equations
• Large Language Models
• Reinforcement Learning
• Verifiable Rewards
• Code Generation

Innovation

• Reinforcement Learning
• Verifiable Physics
• PDE Solvers
• Continuous Rewards
• Code Generation