Learning to reset in target search problems

2025-03-14

AI Summary

This work addresses the dynamic reset strategy optimization problem in target search, proposing a reinforcement learning–based joint learning framework that simultaneously optimizes reset timing and spatial motion decisions (e.g., turning). It is the first to apply deep reinforcement learning—specifically Proximal Policy Optimization (PPO)—to end-to-end co-learning of reset and motion control, enabling adaptive and interpretable policy generation in stochastic environments. The method integrates Brownian motion modeling with stochastic search simulation and employs policy interpretability analysis to uncover environment-dependent, spike-shaped reset distributions. It recovers the theoretical optimal solution on the Brownian reset benchmark and significantly outperforms classical heuristic methods on the joint turning-reset task, demonstrating strong generalization capability and computational efficiency.

Technology Category

• Search and Optimization: Learning to Search
• Intelligent Robots: Learning & Optimization for ROB
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: ML for personalized search and recommendations

Abstract

Target search problems are central to a wide range of fields, from biological foraging to the optimization algorithms. Recently, the ability to reset the search has been shown to significantly improve the searcher's efficiency. However, the optimal resetting strategy depends on the specific properties of the search problem and can often be challenging to determine. In this work, we propose a reinforcement learning (RL)-based framework to train agents capable of optimizing their search efficiency in environments by learning how to reset. First, we validate the approach in a well-established benchmark: the Brownian search with resetting. There, RL agents consistently recover strategies closely resembling the sharp resetting distribution, known to be optimal in this scenario. We then extend the framework by allowing agents to control not only when to reset, but also their spatial dynamics through turning actions. In this more complex setting, the agents discover strategies that adapt both resetting and turning to the properties of the environment, outperforming the proposed benchmarks. These results demonstrate how reinforcement learning can serve both as an optimization tool and a mechanism for uncovering new, interpretable strategies in stochastic search processes with resetting.

Problem

• Optimizing search efficiency using reinforcement learning.
• Determining optimal resetting strategies in target search problems.
• Adapting resetting and spatial dynamics to environmental properties.

Innovation

• Reinforcement learning optimizes search efficiency.
• Agents control reset timing and spatial dynamics.
• RL uncovers interpretable strategies in stochastic searches.