Learn to Change the World: Multi-level Reinforcement Learning with Model-Changing Actions

2025-10-16

AI Summary

Traditional reinforcement learning (RL) agents passively adapt to environmental dynamics, limiting performance in non-stationary settings. Method: We propose a novel paradigm wherein agents actively modify the environment’s dynamics model, formalized via a Multi-layer Configurable Time-Varying Markov Decision Process (MCTVMDP). In this framework, a high-level controller selects “model-modification actions” that reconfigure the transition function of an underlying non-stationary MDP; a low-level policy then executes actions within the modified environment. Both layers are jointly optimized to maximize the expected discounted return. Contribution/Results: This work introduces, for the first time, learnable model-modification actions—enabling proactive environmental restructuring and adaptive dynamical control. Empirical evaluation demonstrates substantial improvements in long-term cumulative reward under non-stationarity, validating that active environment modeling provides fundamental gains in policy performance.

Technology Category

• Multiagent Systems: Multiagent Learning
• Machine Learning: Reinforcement Learning
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Reinforcement learning usually assumes a given or sometimes even fixed environment in which an agent seeks an optimal policy to maximize its long-term discounted reward. In contrast, we consider agents that are not limited to passive adaptations: they instead have model-changing actions that actively modify the RL model of world dynamics itself. Reconfiguring the underlying transition processes can potentially increase the agents' rewards. Motivated by this setting, we introduce the multi-layer configurable time-varying Markov decision process (MCTVMDP). In an MCTVMDP, the lower-level MDP has a non-stationary transition function that is configurable through upper-level model-changing actions. The agent's objective consists of two parts: Optimize the configuration policies in the upper-level MDP and optimize the primitive action policies in the lower-level MDP to jointly improve its expected long-term reward.

Problem

• Agents actively modify world dynamics through model-changing actions
• Introducing multi-layer configurable time-varying Markov decision processes
• Jointly optimizing configuration policies and primitive action policies

Innovation

• Model-changing actions modify world dynamics
• Multi-layer configurable time-varying Markov decision process
• Joint optimization of configuration and primitive action policies