Diffusion Model Predictive Control

2024-10-07

arXiv.org

AI Summary

To address the challenge that model predictive control (MPC) in offline reinforcement learning struggles to adapt to novel reward functions and non-stationary dynamics, this paper proposes Diffusion Model Predictive Control (D-MPC). D-MPC is the first method to unify diffusion models for both multi-step action generation and multi-step dynamics modeling, jointly optimizing both components within online MPC. It enables zero-shot reward reconfiguration and dynamic adaptation, eliminating reliance on pre-specified rewards or fixed dynamics models. Technically, it integrates diffusion-based modeling, uncertainty-aware sequential generation, and model-based planning. On the D4RL benchmark, D-MPC significantly outperforms state-of-the-art model-based offline planning methods such as MBOP, while matching the performance of current top-tier model-based and model-free RL algorithms.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Multiagent Systems: Multiagent Planning
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

We propose Diffusion Model Predictive Control (D-MPC), a novel MPC approach that learns a multi-step action proposal and a multi-step dynamics model, both using diffusion models, and combines them for use in online MPC. On the popular D4RL benchmark, we show performance that is significantly better than existing model-based offline planning methods using MPC (e.g. MBOP) and competitive with state-of-the-art (SOTA) model-based and model-free reinforcement learning methods. We additionally illustrate D-MPC's ability to optimize novel reward functions at run time and adapt to novel dynamics, and highlight its advantages compared to existing diffusion-based planning baselines.

Problem

• Develops Diffusion Model Predictive Control for action and dynamics modeling
• Improves performance over existing model-based offline planning methods
• Enables runtime reward optimization and novel dynamics adaptation

Innovation

• Uses diffusion models for action proposals
• Combines diffusion models with MPC
• Optimizes novel rewards at runtime