Integrating Lagrangian Neural Networks into the Dyna Framework for Reinforcement Learning

2026-03-09

AI Summary

This work proposes a novel approach to model-based reinforcement learning by integrating neural networks with embedded Lagrangian mechanics into the Dyna framework, addressing the limitations of conventional black-box dynamics models that often fail to generalize to out-of-distribution data and violate physical laws, leading to prediction errors. By explicitly encoding physical priors into the model architecture, the method enhances both generalization capability and sample efficiency. During training, a hybrid optimization strategy combines stochastic gradient descent with a state estimation optimizer, the latter of which substantially accelerates convergence. Empirical results from simulation experiments demonstrate that the proposed approach achieves more accurate predictions and more efficient learning in out-of-distribution scenarios, underscoring the critical role of physics-informed structural priors in improving model performance.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Reinforcement Learning
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Model-based reinforcement learning (MBRL) is sample-efficient but depends on the accuracy of the learned dynamics, which are often modeled using black-box methods that do not adhere to physical laws. Those methods tend to produce inaccurate predictions when presented with data that differ from the original training set. In this work, we employ Lagrangian neural networks (LNNs), which enforce an underlying Lagrangian structure to train the model within a Dyna-based MBRL framework. Furthermore, we train the LNN using stochastic gradient-based and state-estimation-based optimizers to learn the network's weights. The state-estimation-based method converges faster than the stochastic gradient-based method during neural network training. Simulation results are provided to illustrate the effectiveness of the proposed LNN-based Dyna framework for MBRL.

Problem

• model-based reinforcement learning
• dynamics modeling
• physical laws
• generalization
• black-box models

Innovation

• Lagrangian Neural Networks
• Model-based Reinforcement Learning
• Dyna Framework
• Physical Structure Preservation
• State Estimation Optimization