When Should a Robot Replan? Regret-Guided Update Scheduling in Time-Varying MDPs

2026-06-15

AI Summary

This work addresses the challenge of efficient decision-making for resource-constrained robots operating in non-stationary environments, where frequent replanning is often infeasible due to limited energy and computational budgets. Focusing on time-varying Markov decision processes (TVMDPs), the authors propose an adaptive update-skipping mechanism that employs maximum likelihood estimation to monitor environmental dynamics and triggers costly policy replanning only when necessary. During periods of stability, the method reuses the existing policy while propagating state predictions. The key theoretical contribution is the first derivation of a dynamic regret bound that explicitly links environmental drift rate, update intervals, and cumulative regret, enabling intelligent allocation of computational resources. Empirical evaluations on a Mars rover slip navigation simulator and real-world Crazyflie quadrotor obstacle avoidance tasks demonstrate significant performance improvements over existing budget-aware baselines.

Abstract

Robots operating in non-stationary environments must continually adapt their policies as the dynamics drift, but onboard energy and compute budgets cap how often a full state estimation and re-planning step can be performed. This raises a question: \emph{when}, along a horizon, should a robot spend its limited budget? We formulate this problem in time-varying Markov decision processes (TVMDPs) with a known bound on the rate of transition drift. We model execution as a \emph{skip-update} scheme in which, at chosen update times, the agent estimates the transition kernel by maximum likelihood and computes a finite-horizon policy, and between updates reuses this policy under a propagated state estimate. We analyze the dynamic regret of this scheme and show how it grows during skip intervals in terms of the properties of the TVMDP and the skip lengths; the resulting bound answers the opening question via an online, regret-guided update rule that allocates the budget adaptively. We evaluate the rule in a simulated Mars-rover navigation task with time-varying slip dynamics and on a Crazyflie quadrotor in indoor obstacle fields. Adaptive allocation outperforms other budgeted baselines.

Problem

• time-varying MDPs
• regret
• replanning
• non-stationary environments
• update scheduling

Innovation

• regret-guided replanning
• time-varying MDPs
• adaptive update scheduling
• skip-update scheme
• dynamic regret