Learning to Search for Vehicle Routing with Multiple Time Windows

2025-05-29

AI Summary

This paper addresses the Vehicle Routing Problem with Multiple Time Windows (VRPMTW). We propose a Reinforcement Learning-driven Adaptive Variable Neighborhood Search (RL-AVNS) algorithm. Methodologically, we introduce— for the first time—a RL-based operator selection mechanism that jointly leverages real-time solution states and historical experience; incorporate a customer-level time-flexibility metric to enhance perturbation strategies; and design a Transformer-based policy network to guide intelligent local search. Compared against conventional VNS, classical AVNS, and state-of-the-art learning-enhanced heuristics, RL-AVNS achieves significant improvements in solution quality (3.2%–7.8% average reduction in total cost) on real-world vending machine replenishment instances. The algorithm demonstrates strong generalization capability and scalability, efficiently solving large-scale, highly constrained VRPMTW instances. It establishes an extensible, intelligent optimization paradigm for dynamic logistics scheduling.

Technology Category

• Planning, Routing, and Scheduling: Learning for Planning and Scheduling
• Search and Optimization: Learning to Search
• Machine Learning: Reinforcement Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

In this study, we propose a reinforcement learning-based adaptive variable neighborhood search (RL-AVNS) method designed for effectively solving the Vehicle Routing Problem with Multiple Time Windows (VRPMTW). Unlike traditional adaptive approaches that rely solely on historical operator performance, our method integrates a reinforcement learning framework to dynamically select neighborhood operators based on real-time solution states and learned experience. We introduce a fitness metric that quantifies customers' temporal flexibility to improve the shaking phase, and employ a transformer-based neural policy network to intelligently guide operator selection during the local search. Extensive computational experiments are conducted on realistic scenarios derived from the replenishment of unmanned vending machines, characterized by multiple clustered replenishment windows. Results demonstrate that RL-AVNS significantly outperforms traditional variable neighborhood search (VNS), adaptive VNS (AVNS), and state-of-the-art learning-based heuristics, achieving substantial improvements in solution quality and computational efficiency across various instance scales and time window complexities. Particularly notable is the algorithm's capability to generalize effectively to problem instances not encountered during training, underscoring its practical utility for complex logistics scenarios.

Problem

• Solves Vehicle Routing Problem with Multiple Time Windows (VRPMTW)
• Dynamically selects neighborhood operators via reinforcement learning
• Improves solution quality and efficiency for complex logistics

Innovation

• Reinforcement learning-based adaptive variable neighborhood search
• Fitness metric for temporal flexibility in shaking
• Transformer-based neural policy for operator selection